[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:28 dist-test-slave-qnz0 krb5kdc[26099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:28 dist-test-slave-qnz0 krb5kdc[26099](info): set up 2 sockets
Sep 28 15:02:28 dist-test-slave-qnz0 krb5kdc[26099](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 15:02:29.463200 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.359s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:29 dist-test-slave-qnz0 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:29.634142 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:46177
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:29.743871 26112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:29.744199 26112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:29.744241 26112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:29.746042 26112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:29.746138 26112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:29.746165 26112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:29.746186 26112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:29.746204 26112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:29.748613 26112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:46177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:46177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:29.749126 26112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:29.749459 26112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:29.756233 26120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:29.756276 26118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:29.756247 26117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:29.756649 26112 server_base.cc:1034] running on GCE node
I20260928 15:02:29.757167 26112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:29.757632 26112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:29.758899 26112 hybrid_clock.cc:648] HybridClock initialized: now 1790607749758865 us; error 51 us; skew 500 ppm
Sep 28 15:02:29 dist-test-slave-qnz0 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:29.762617 26112 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:29.763350 26112 webserver.cc:460] Webserver started at http://127.25.123.62:39055/ using document root <none> and password file <none>
I20260928 15:02:29.763639 26112 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:29.763716 26112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:29.763942 26112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:29.765221 26112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "ee411b9b2c834ded80e0e5ab10363156"
format_stamp: "Formatted at 2026-09-28 15:02:29 on dist-test-slave-qnz0"
server_key: "2312eb1f2ad26bae5ad5402b5a11cb93"
server_key_iv: "b8e31ced91ca13c1fc8bc8b5facea815"
server_key_version: "encryptionkey@0"
I20260928 15:02:29.765625 26112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "ee411b9b2c834ded80e0e5ab10363156"
format_stamp: "Formatted at 2026-09-28 15:02:29 on dist-test-slave-qnz0"
server_key: "2312eb1f2ad26bae5ad5402b5a11cb93"
server_key_iv: "b8e31ced91ca13c1fc8bc8b5facea815"
server_key_version: "encryptionkey@0"
I20260928 15:02:29.767455 26112 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:02:29.768996 26127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:29.769709 26112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:29.770047 26112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "ee411b9b2c834ded80e0e5ab10363156"
format_stamp: "Formatted at 2026-09-28 15:02:29 on dist-test-slave-qnz0"
server_key: "2312eb1f2ad26bae5ad5402b5a11cb93"
server_key_iv: "b8e31ced91ca13c1fc8bc8b5facea815"
server_key_version: "encryptionkey@0"
I20260928 15:02:29.770273 26112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:29.794632 26112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:29.798606 26112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:29.798858 26112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:29.805547 26112 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:46177
I20260928 15:02:29.805733 26179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:46177 every 8 connection(s)
I20260928 15:02:29.806174 26112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:29.807557 26180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:29.811184 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156: Bootstrap starting.
I20260928 15:02:29.812155 26180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:29.812373 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26112
I20260928 15:02:29.812543 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:29.812683 26180 log.cc:824] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:29.813387 26092 external_mini_cluster.cc:1428] Setting key 0938c13500f8418470ff6a01703be1b9
I20260928 15:02:29.813719 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156: No bootstrap required, opened a new log
I20260928 15:02:29.815999 26180 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee411b9b2c834ded80e0e5ab10363156" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46177 } }
I20260928 15:02:29.816183 26180 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:29.816272 26180 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee411b9b2c834ded80e0e5ab10363156, State: Initialized, Role: FOLLOWER
I20260928 15:02:29.816540 26180 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee411b9b2c834ded80e0e5ab10363156" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46177 } }
I20260928 15:02:29.816633 26180 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:29.816669 26180 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:29.816716 26180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:29.818301 26180 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee411b9b2c834ded80e0e5ab10363156" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46177 } }
I20260928 15:02:29.818414 26180 leader_election.cc:302] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee411b9b2c834ded80e0e5ab10363156; no voters:
I20260928 15:02:29.818692 26180 leader_election.cc:288] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:29.818763 26185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:29.818909 26185 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [term 1 LEADER]: Becoming Leader. State: Replica: ee411b9b2c834ded80e0e5ab10363156, State: Running, Role: LEADER
I20260928 15:02:29.819087 26185 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee411b9b2c834ded80e0e5ab10363156" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46177 } }
I20260928 15:02:29.819128 26180 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:29.819527 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee411b9b2c834ded80e0e5ab10363156. Latest consensus state: current_term: 1 leader_uuid: "ee411b9b2c834ded80e0e5ab10363156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee411b9b2c834ded80e0e5ab10363156" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46177 } } }
I20260928 15:02:29.819540 26187 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee411b9b2c834ded80e0e5ab10363156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee411b9b2c834ded80e0e5ab10363156" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46177 } } }
I20260928 15:02:29.819628 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:29.819646 26187 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:29.820515 26191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:29.821342 26191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:29.824015 26191 catalog_manager.cc:1347] Generated new cluster ID: a8671d407ae942b78da1cf82ba943fea
I20260928 15:02:29.824095 26191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:02:29 dist-test-slave-qnz0 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:29.837328 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:29.814436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.25.123.62:46177)
0928 15:02:29.814628 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:29.814638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:29.815067 (+ 429us) server_negotiation.cc:407] Connection header received
0928 15:02:29.817433 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:29.817439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:29.817490 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:29.817657 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:29.820265 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.822122 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.823970 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:29.824812 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:29.824946 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:29.831697 (+ 6751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:29.831704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:29.831704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:29.831721 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:29.834525 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.835223 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.835227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.835228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.835377 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:29.835793 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:29.835793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:29.835794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:29.836124 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:29.836274 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:29.836758 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 15:02:29.836910 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260928 15:02:29.852077 26191 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:29.852914 26191 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:29.858474 26191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee411b9b2c834ded80e0e5ab10363156: Generated new TSK 0
I20260928 15:02:29.858727 26191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:29.907260 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:46177
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:30.079314 26206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:30.079628 26206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:30.079669 26206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:30.082540 26206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:30.082634 26206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:30.082739 26206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:30.085206 26206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:46177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:30.085507 26206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:30.085728 26206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:30.096849 26212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:30.096902 26211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:30.097142 26214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:30.097301 26206 server_base.cc:1034] running on GCE node
I20260928 15:02:30.097872 26206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:30.098493 26206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:30.100069 26206 hybrid_clock.cc:648] HybridClock initialized: now 1790607750099756 us; error 145 us; skew 500 ppm
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:30.103211 26206 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:30.103893 26206 webserver.cc:460] Webserver started at http://127.25.123.1:44099/ using document root <none> and password file <none>
I20260928 15:02:30.104168 26206 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:30.104239 26206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:30.104426 26206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:30.105775 26206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "5a91ee7f60f440299b4e48ba20d8d13d"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "ff1f5a17883b27797516a1317187219f"
server_key_iv: "4ea8e670bfa58da4b335eae1a676beb5"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.106253 26206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "5a91ee7f60f440299b4e48ba20d8d13d"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "ff1f5a17883b27797516a1317187219f"
server_key_iv: "4ea8e670bfa58da4b335eae1a676beb5"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.108796 26206 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 15:02:30.111636 26221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.112630 26206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:02:30.112779 26206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "5a91ee7f60f440299b4e48ba20d8d13d"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "ff1f5a17883b27797516a1317187219f"
server_key_iv: "4ea8e670bfa58da4b335eae1a676beb5"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.112922 26206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:30.148049 26206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:30.156579 26206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:30.157143 26206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:30.157680 26206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:30.158228 26206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:30.158291 26206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.158339 26206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:30.158367 26206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.167511 26206 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:34491
I20260928 15:02:30.167635 26334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:34491 every 8 connection(s)
I20260928 15:02:30.167974 26206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:30.168319 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26206
I20260928 15:02:30.168447 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:30.168602 26092 external_mini_cluster.cc:1428] Setting key d535703da2110d535f3c8b1b5bad0bb5
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:02:30.182802 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.169095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:38181 (local address 127.25.123.62:46177)
0928 15:02:30.169173 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:30.169176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:30.170071 (+ 895us) server_negotiation.cc:407] Connection header received
0928 15:02:30.171025 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:30.171027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:30.171048 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:30.171102 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:30.172878 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.173461 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.174647 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.174942 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.175028 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.178189 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:30.178192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:30.178192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:30.178202 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:30.180953 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.181605 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.181607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.181608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.181680 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.182008 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.182010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.182010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.182158 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:30.182246 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:30.182573 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 15:02:30.182689 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:02:30.183046 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.169150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:46177 (local address 127.25.123.1:38181)
0928 15:02:30.169798 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:30.169883 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:30.170690 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:30.171250 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:30.171256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:30.172359 (+ 1103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:30.172730 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:30.172740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.173580 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.173582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:30.174553 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:30.174557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.175038 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.175039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:30.175269 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.176021 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:30.176031 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:30.177987 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:30.181097 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:30.181111 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:30.181114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:30.181493 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:30.181758 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:30.181760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:30.181760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:30.181927 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:30.182236 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:30.182243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:30.182415 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 15:02:30.182548 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 15:02:30.182764 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":248,"threads_started":1}
I20260928 15:02:30.184047 26335 heartbeater.cc:347] Connected to a master server at 127.25.123.62:46177
I20260928 15:02:30.184154 26335 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:30.184373 26335 heartbeater.cc:511] Master 127.25.123.62:46177 requested a full tablet report, sending...
I20260928 15:02:30.185504 26144 ts_manager.cc:194] Registered new tserver with Master: 5a91ee7f60f440299b4e48ba20d8d13d (127.25.123.1:34491)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
I20260928 15:02:30.186343 26144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:38181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:30.214594 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:46177
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:30.325106 26340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:30.325461 26340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:30.325528 26340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:30.327076 26340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:30.327188 26340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:30.327246 26340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:30.329172 26340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:46177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:30.329732 26340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:30.330206 26340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:30.341612 26345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:30.341686 26346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:30.341737 26340 server_base.cc:1034] running on GCE node
W20260928 15:02:30.341682 26348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:30.342335 26340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:30.342674 26340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:30.343966 26340 hybrid_clock.cc:648] HybridClock initialized: now 1790607750343941 us; error 47 us; skew 500 ppm
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:30.347461 26340 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:30.348063 26340 webserver.cc:460] Webserver started at http://127.25.123.2:34867/ using document root <none> and password file <none>
I20260928 15:02:30.348317 26340 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:30.348423 26340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:30.348601 26340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:30.350351 26340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "13e9df65305d4926b3cd1bf450f23fd1"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "b84013e82cd4060aa61148c167f86264"
server_key_iv: "07b8ce4ce3b05431f3ab11e73048abab"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.351007 26340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "13e9df65305d4926b3cd1bf450f23fd1"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "b84013e82cd4060aa61148c167f86264"
server_key_iv: "07b8ce4ce3b05431f3ab11e73048abab"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.353122 26340 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 15:02:30.354409 26355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.354812 26340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:30.354966 26340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "13e9df65305d4926b3cd1bf450f23fd1"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "b84013e82cd4060aa61148c167f86264"
server_key_iv: "07b8ce4ce3b05431f3ab11e73048abab"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.355104 26340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:30.390828 26340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:30.393959 26340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:30.394187 26340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:30.394474 26340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:30.394829 26340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:30.394889 26340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.394932 26340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:30.394961 26340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.406801 26340 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:46423
I20260928 15:02:30.406879 26468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:46423 every 8 connection(s)
I20260928 15:02:30.407327 26340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:30.414636 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26340
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:30.415022 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:30.415830 26092 external_mini_cluster.cc:1428] Setting key 926a39c206fe2c208c3b62eb4dd2484e
I20260928 15:02:30.425897 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.408225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:36237 (local address 127.25.123.62:46177)
0928 15:02:30.408335 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:30.408338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:30.408692 (+ 354us) server_negotiation.cc:407] Connection header received
0928 15:02:30.409592 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:30.409594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:30.409607 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:30.409674 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:30.410861 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.411380 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.412799 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.413027 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.413102 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.415884 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:30.415886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:30.415887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:30.415895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:30.418524 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.419262 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.419264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.419265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.419368 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.419784 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.419785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.419785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.420022 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:30.420149 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:30.425438 (+ 5289us) server_negotiation.cc:299] Negotiation successful
0928 15:02:30.425623 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:02:30.425961 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.408177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:46177 (local address 127.25.123.2:36237)
0928 15:02:30.408456 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:30.408521 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:30.409400 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:30.409751 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:30.409755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:30.410437 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:30.410656 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:30.410662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.411569 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.411571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:30.412663 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:30.412668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.413194 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.413195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:30.413361 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.414058 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:30.414067 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:30.415730 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:30.418630 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:30.418644 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:30.418647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:30.419080 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:30.419486 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:30.419488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:30.419488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:30.419697 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:30.424631 (+ 4934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:30.424651 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:30.425121 (+ 470us) client_negotiation.cc:769] Sending connection context
0928 15:02:30.425323 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 15:02:30.425590 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":129,"threads_started":1}
I20260928 15:02:30.426591 26469 heartbeater.cc:347] Connected to a master server at 127.25.123.62:46177
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:02:30.426713 26469 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:30.426919 26469 heartbeater.cc:511] Master 127.25.123.62:46177 requested a full tablet report, sending...
I20260928 15:02:30.427577 26144 ts_manager.cc:194] Registered new tserver with Master: 13e9df65305d4926b3cd1bf450f23fd1 (127.25.123.2:46423)
I20260928 15:02:30.428083 26144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:36237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:30.460636 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:46177
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:30.574529 26474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:30.574839 26474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:30.574877 26474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:30.577001 26474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:30.577072 26474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:30.577111 26474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:30.579272 26474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:46177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:30.579586 26474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:30.579903 26474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:30.587265 26482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:30.587299 26480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:30.587265 26479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:30.588016 26474 server_base.cc:1034] running on GCE node
I20260928 15:02:30.588299 26474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:30.588639 26474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:30.589849 26474 hybrid_clock.cc:648] HybridClock initialized: now 1790607750589801 us; error 77 us; skew 500 ppm
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:30.603543 26474 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:30.604254 26474 webserver.cc:460] Webserver started at http://127.25.123.3:34379/ using document root <none> and password file <none>
I20260928 15:02:30.604576 26474 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:30.604652 26474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:30.604826 26474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:30.606010 26474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "4e4b12da409144828b4f848954087761"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "cd2fbc6d10e160350a384fea72c70405"
server_key_iv: "715c39e58969151516a617e51c15278f"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.606410 26474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "4e4b12da409144828b4f848954087761"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "cd2fbc6d10e160350a384fea72c70405"
server_key_iv: "715c39e58969151516a617e51c15278f"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.607921 26474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:02:30.609041 26489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.609377 26474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:02:30.609509 26474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "4e4b12da409144828b4f848954087761"
format_stamp: "Formatted at 2026-09-28 15:02:30 on dist-test-slave-qnz0"
server_key: "cd2fbc6d10e160350a384fea72c70405"
server_key_iv: "715c39e58969151516a617e51c15278f"
server_key_version: "encryptionkey@0"
I20260928 15:02:30.609592 26474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:30.623104 26474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:30.626528 26474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:30.626935 26474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:30.627375 26474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:30.627767 26474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:30.627830 26474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.627875 26474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:30.627902 26474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:30.638267 26474 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:35625
I20260928 15:02:30.638306 26602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:35625 every 8 connection(s)
I20260928 15:02:30.638680 26474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:30.648916 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26474
I20260928 15:02:30.649050 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:30.649232 26092 external_mini_cluster.cc:1428] Setting key e70596473acb4a1f201265c058ed2e2f
I20260928 15:02:30.651072 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.639619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:39103 (local address 127.25.123.62:46177)
0928 15:02:30.639716 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:30.639718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:30.640047 (+ 329us) server_negotiation.cc:407] Connection header received
0928 15:02:30.640688 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:30.640689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:30.640702 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:30.640769 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:30.641820 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.642339 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.643427 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.643632 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.643728 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.646483 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:30.646485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:30.646485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:30.646493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:30.648466 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.648993 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.648995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.648995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.649071 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.649436 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.649437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.649438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.649654 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:30.649727 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:30.650820 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0928 15:02:30.650972 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:02:30.651605 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.639579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:46177 (local address 127.25.123.3:39103)
0928 15:02:30.639807 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:30.639936 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:30.640573 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:30.640852 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:30.640856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:30.641475 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:30.641688 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:30.641693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.642443 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.642444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:30.643279 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:30.643281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.643743 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.643744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:30.643884 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.644647 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:30.644657 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:30.646342 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:30.648566 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:30.648576 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:30.648578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:30.648898 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:30.649148 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:30.649149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:30.649149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:30.649350 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:30.649744 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:30.649752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:30.650139 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 15:02:30.650765 (+ 626us) client_negotiation.cc:240] Negotiation successful
0928 15:02:30.651235 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260928 15:02:30.652561 26603 heartbeater.cc:347] Connected to a master server at 127.25.123.62:46177
I20260928 15:02:30.652683 26603 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:30.652952 26603 heartbeater.cc:511] Master 127.25.123.62:46177 requested a full tablet report, sending...
I20260928 15:02:30.653715 26144 ts_manager.cc:194] Registered new tserver with Master: 4e4b12da409144828b4f848954087761 (127.25.123.3:35625)
I20260928 15:02:30.654165 26144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:39103
I20260928 15:02:30.664055 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:02:30.696501 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.685466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.25.123.1:34491)
0928 15:02:30.685609 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:30.685621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:30.685675 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:02:30.685702 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:30.685704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:30.685719 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:30.685791 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:30.686570 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.687369 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.688223 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.688679 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.688756 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.692602 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:30.692619 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:30.692620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:30.692632 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:30.694980 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.695396 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.695399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.695399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.695458 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.695809 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.695810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.695810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.695967 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:30.696084 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:30.696237 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 15:02:30.696335 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
W20260928 15:02:30.696969 26248 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41952
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:02:30.732849 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:30.718574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.25.123.1:34491)
0928 15:02:30.718921 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:30.718924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:30.719232 (+ 308us) server_negotiation.cc:407] Connection header received
0928 15:02:30.719265 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:30.719265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:30.719280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:30.719352 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:30.721198 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.721756 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.722973 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:30.723184 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:30.723253 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:30.727395 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:30.727397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:30.727397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:30.727406 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:30.729960 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.730634 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.730635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.730636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.730725 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:30.731399 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:30.731401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:30.731402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:30.731606 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:30.731734 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:30.732034 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 15:02:30.732304 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260928 15:02:30.735163 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26206
I20260928 15:02:30.745467 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26340
I20260928 15:02:30.751300 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26474
I20260928 15:02:30.756884 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26112
2026-09-28T15:02:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2689 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26617](info): set up 2 sockets
Sep 28 15:02:30 dist-test-slave-qnz0 krb5kdc[26617](info): commencing operation
krb5kdc: starting...
W20260928 15:02:32.012701 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:32 dist-test-slave-qnz0 krb5kdc[26617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:32.170565 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35731
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:32.282529 26630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:32.282769 26630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:32.282796 26630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:32.284162 26630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:32.284214 26630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:32.284229 26630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:32.284242 26630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:32.284255 26630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:32.286094 26630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:32.286432 26630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:32.286726 26630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:32.294021 26636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.294021 26638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.295174 26635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:32.295475 26630 server_base.cc:1034] running on GCE node
I20260928 15:02:32.295763 26630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:32.296116 26630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:32.297334 26630 hybrid_clock.cc:648] HybridClock initialized: now 1790607752297298 us; error 74 us; skew 500 ppm
Sep 28 15:02:32 dist-test-slave-qnz0 krb5kdc[26617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:32.301275 26630 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:32.301869 26630 webserver.cc:460] Webserver started at http://127.25.123.62:39897/ using document root <none> and password file <none>
I20260928 15:02:32.302152 26630 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:32.302225 26630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:32.302374 26630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:32.303915 26630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "0adab512a19d42038af9f405aaa92fb7"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "f9ed4aeb14b8f6b4009b2044233a937f"
server_key_iv: "41b0cea7c6d55b8efefbc96749d8acf0"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.304368 26630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "0adab512a19d42038af9f405aaa92fb7"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "f9ed4aeb14b8f6b4009b2044233a937f"
server_key_iv: "41b0cea7c6d55b8efefbc96749d8acf0"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.306682 26630 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:32.307983 26645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.308295 26630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:32.308660 26630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "0adab512a19d42038af9f405aaa92fb7"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "f9ed4aeb14b8f6b4009b2044233a937f"
server_key_iv: "41b0cea7c6d55b8efefbc96749d8acf0"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.308782 26630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:32.321477 26630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:32.324607 26630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:32.324997 26630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:32.329741 26630 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:35731
I20260928 15:02:32.330190 26630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:32.330168 26697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:35731 every 8 connection(s)
I20260928 15:02:32.331249 26698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:32.334604 26698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7: Bootstrap starting.
I20260928 15:02:32.336000 26698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:32.336352 26698 log.cc:824] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:32.337786 26698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7: No bootstrap required, opened a new log
I20260928 15:02:32.339040 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26630
I20260928 15:02:32.339251 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:32.339509 26092 external_mini_cluster.cc:1428] Setting key d3c760c13e92dc9e2ab10a6e0910b955
I20260928 15:02:32.340795 26698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adab512a19d42038af9f405aaa92fb7" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35731 } }
I20260928 15:02:32.341065 26698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:32.341101 26698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0adab512a19d42038af9f405aaa92fb7, State: Initialized, Role: FOLLOWER
I20260928 15:02:32.341239 26698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adab512a19d42038af9f405aaa92fb7" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35731 } }
I20260928 15:02:32.341357 26698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:32.341393 26698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:32.341436 26698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:32.342265 26698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adab512a19d42038af9f405aaa92fb7" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35731 } }
I20260928 15:02:32.342382 26698 leader_election.cc:302] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0adab512a19d42038af9f405aaa92fb7; no voters:
I20260928 15:02:32.342566 26698 leader_election.cc:288] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:32.342731 26703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:32.342873 26698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:32.342990 26703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 0adab512a19d42038af9f405aaa92fb7, State: Running, Role: LEADER
I20260928 15:02:32.343232 26703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adab512a19d42038af9f405aaa92fb7" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35731 } }
I20260928 15:02:32.343775 26703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0adab512a19d42038af9f405aaa92fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adab512a19d42038af9f405aaa92fb7" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35731 } } }
I20260928 15:02:32.343945 26703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:32.344202 26704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0adab512a19d42038af9f405aaa92fb7. Latest consensus state: current_term: 1 leader_uuid: "0adab512a19d42038af9f405aaa92fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adab512a19d42038af9f405aaa92fb7" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35731 } } }
I20260928 15:02:32.344455 26704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:32.345603 26717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:32.345726 26717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:32.345981 26718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:32.346210 26718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:32.347911 26718 catalog_manager.cc:1347] Generated new cluster ID: d18df53e05fc4cf7a498be4c9b3b4281
I20260928 15:02:32.347982 26718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:02:32 dist-test-slave-qnz0 krb5kdc[26617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:32.356189 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.340648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.25.123.62:35731)
0928 15:02:32.340944 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:32.340956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:32.340997 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:02:32.341125 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:32.341134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:32.341197 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:32.341408 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:32.342267 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.343703 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.344877 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.345531 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.345652 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.348982 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:32.348987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:32.348987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:32.349002 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:32.351944 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.352539 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.352544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.352545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.352682 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.353490 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.353493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.353494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.354141 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:32.354371 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:32.354910 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 15:02:32.355554 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":143,"threads_started":1}
I20260928 15:02:32.398123 26718 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:32.398957 26718 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:32.406255 26718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0adab512a19d42038af9f405aaa92fb7: Generated new TSK 0
I20260928 15:02:32.406540 26718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:32.462858 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35731
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:32.614521 26724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:32.614806 26724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:32.614852 26724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:32.616282 26724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:32.616364 26724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:32.616439 26724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:32.619901 26724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:32.620296 26724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:32.620688 26724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:32.626782 26732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.627065 26729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.627302 26730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:32.627331 26724 server_base.cc:1034] running on GCE node
I20260928 15:02:32.627763 26724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:32.628072 26724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:32.629277 26724 hybrid_clock.cc:648] HybridClock initialized: now 1790607752629258 us; error 80 us; skew 500 ppm
Sep 28 15:02:32 dist-test-slave-qnz0 krb5kdc[26617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:32.636747 26724 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:32.637456 26724 webserver.cc:460] Webserver started at http://127.25.123.1:38821/ using document root <none> and password file <none>
I20260928 15:02:32.637823 26724 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:32.637940 26724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:32.638114 26724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:32.639456 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "d29a79de06e142a9aa5c61eefec379b1"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "77a47b1bcb31366ed7e989910762124a"
server_key_iv: "c992c4fdce275b0106587f5966f57809"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.640215 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "d29a79de06e142a9aa5c61eefec379b1"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "77a47b1bcb31366ed7e989910762124a"
server_key_iv: "c992c4fdce275b0106587f5966f57809"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.642242 26724 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:02:32.643402 26739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.643641 26724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:32.643757 26724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "d29a79de06e142a9aa5c61eefec379b1"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "77a47b1bcb31366ed7e989910762124a"
server_key_iv: "c992c4fdce275b0106587f5966f57809"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.643864 26724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:32.662345 26724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:32.665484 26724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:32.665650 26724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:32.665926 26724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:32.666334 26724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:32.666385 26724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.666414 26724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:32.666427 26724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.676682 26724 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:39587
I20260928 15:02:32.676780 26852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:39587 every 8 connection(s)
I20260928 15:02:32.677131 26724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:32.684437 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26724
I20260928 15:02:32.684841 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:32.685130 26092 external_mini_cluster.cc:1428] Setting key 5d8e5131e11b1c44fdc3a3bb2d483860
Sep 28 15:02:32 dist-test-slave-qnz0 krb5kdc[26617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:32.693873 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.678718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:57729 (local address 127.25.123.62:35731)
0928 15:02:32.678856 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:32.678859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:32.679360 (+ 501us) server_negotiation.cc:407] Connection header received
0928 15:02:32.681026 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:32.681028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:32.681062 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:32.681127 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:32.681918 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.682484 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.684082 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.684368 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.684818 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.688108 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:32.688110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:32.688111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:32.688123 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:32.690995 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.691840 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.691842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.691843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.691924 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.692681 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.692683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.692683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.692885 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:32.692992 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:32.693602 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 15:02:32.693725 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 15:02:32.694919 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.678785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35731 (local address 127.25.123.1:57729)
0928 15:02:32.679077 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:32.679125 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:32.680872 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:32.681216 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:32.681221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:32.681551 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:32.681793 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.681798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.682612 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.682614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.683840 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.683844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.684827 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.684829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.684977 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.685735 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:32.685745 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:32.687886 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:32.691374 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.691398 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.691400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.691722 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.692245 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.692246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.692247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.692383 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.693056 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:32.693072 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:32.693339 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 15:02:32.693544 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 15:02:32.694143 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":159,"threads_started":1}
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:02:32.696944 26853 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35731
I20260928 15:02:32.697069 26853 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:32.697293 26853 heartbeater.cc:511] Master 127.25.123.62:35731 requested a full tablet report, sending...
I20260928 15:02:32.698873 26662 ts_manager.cc:194] Registered new tserver with Master: d29a79de06e142a9aa5c61eefec379b1 (127.25.123.1:39587)
I20260928 15:02:32.699847 26662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:57729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:32.726323 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35731
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:32.831087 26858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:32.831336 26858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:32.831365 26858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:32.833756 26858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:32.833845 26858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:32.833887 26858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:32.839505 26858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:32.839847 26858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:32.840142 26858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:32.847941 26863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.847966 26866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:32.848193 26864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:32.848918 26858 server_base.cc:1034] running on GCE node
I20260928 15:02:32.849479 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:32.849952 26858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:32.851238 26858 hybrid_clock.cc:648] HybridClock initialized: now 1790607752851143 us; error 51 us; skew 500 ppm
Sep 28 15:02:32 dist-test-slave-qnz0 krb5kdc[26617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:32.855648 26858 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:32.856341 26858 webserver.cc:460] Webserver started at http://127.25.123.2:46161/ using document root <none> and password file <none>
I20260928 15:02:32.856637 26858 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:32.856689 26858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:32.856833 26858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:32.857973 26858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "ce2da5821dc440f88ed5611fe4958e39"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "97a7ccdc8ff3b5e5772422076cff939a"
server_key_iv: "e15a143b259ea69d7ff961f16e7f3df0"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.858374 26858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "ce2da5821dc440f88ed5611fe4958e39"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "97a7ccdc8ff3b5e5772422076cff939a"
server_key_iv: "e15a143b259ea69d7ff961f16e7f3df0"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.860361 26858 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:02:32.862018 26873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.862447 26858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:32.862572 26858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "ce2da5821dc440f88ed5611fe4958e39"
format_stamp: "Formatted at 2026-09-28 15:02:32 on dist-test-slave-qnz0"
server_key: "97a7ccdc8ff3b5e5772422076cff939a"
server_key_iv: "e15a143b259ea69d7ff961f16e7f3df0"
server_key_version: "encryptionkey@0"
I20260928 15:02:32.862641 26858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:32.877112 26858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:32.879899 26858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:32.880038 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:32.880249 26858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:32.880863 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:32.880918 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.880946 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:32.880965 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:32.890537 26858 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:45567
I20260928 15:02:32.890594 26986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:45567 every 8 connection(s)
I20260928 15:02:32.891405 26858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:32.894220 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26858
I20260928 15:02:32.894462 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:32.894917 26092 external_mini_cluster.cc:1428] Setting key bd8de6f6a5d99fcf5d0e082d46d5b9b0
Sep 28 15:02:32 dist-test-slave-qnz0 krb5kdc[26617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:02:32.908915 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.892567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:53849 (local address 127.25.123.62:35731)
0928 15:02:32.892658 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:32.892660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:32.893062 (+ 402us) server_negotiation.cc:407] Connection header received
0928 15:02:32.894499 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:32.894502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:32.894525 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:32.894824 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:32.896062 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.896933 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.899628 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.899916 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.900001 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.903752 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:32.903761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:32.903761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:32.903772 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:32.905961 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.907010 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.907012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.907013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.907124 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:32.907865 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:32.907867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:32.907867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:32.908151 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:32.908268 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:32.908737 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 15:02:32.908835 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:02:32.909081 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:32.892572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35731 (local address 127.25.123.2:53849)
0928 15:02:32.892749 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:32.892792 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:32.894221 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:32.895028 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:32.895034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:32.895358 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:32.895680 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.895688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.897107 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.897108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.899434 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:32.899440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:32.900035 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:32.900036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:32.900225 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:32.901002 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:32.901015 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:32.903543 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:32.906106 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.906122 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.906126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.906870 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.907216 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:32.907217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:32.907217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:32.907493 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:32.908266 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:32.908278 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:32.908543 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 15:02:32.908669 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 15:02:32.908853 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260928 15:02:32.910385 26987 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35731
I20260928 15:02:32.910502 26987 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:32.910725 26987 heartbeater.cc:511] Master 127.25.123.62:35731 requested a full tablet report, sending...
I20260928 15:02:32.911301 26662 ts_manager.cc:194] Registered new tserver with Master: ce2da5821dc440f88ed5611fe4958e39 (127.25.123.2:45567)
I20260928 15:02:32.911859 26662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:53849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:32.944377 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35731
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:33.071571 26992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:33.071857 26992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:33.071890 26992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:33.073354 26992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:33.073429 26992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:33.073462 26992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:33.075917 26992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:33.076314 26992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:33.076735 26992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:33.083819 27000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:33.084168 26992 server_base.cc:1034] running on GCE node
W20260928 15:02:33.084065 26998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:33.084159 26997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:33.084662 26992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:33.084983 26992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:33.086334 26992 hybrid_clock.cc:648] HybridClock initialized: now 1790607753086278 us; error 58 us; skew 500 ppm
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[26617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:33.090878 26992 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:33.091905 26992 webserver.cc:460] Webserver started at http://127.25.123.3:46263/ using document root <none> and password file <none>
I20260928 15:02:33.092964 26992 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:33.093048 26992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:33.093219 26992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:33.095602 26992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "880b3a9a93ac43e59c6c7385fc01844a"
format_stamp: "Formatted at 2026-09-28 15:02:33 on dist-test-slave-qnz0"
server_key: "484f0d2ac2d96446233db8efba2352d0"
server_key_iv: "9a052fe5d225911c9142c4d059ace71d"
server_key_version: "encryptionkey@0"
I20260928 15:02:33.096717 26992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "880b3a9a93ac43e59c6c7385fc01844a"
format_stamp: "Formatted at 2026-09-28 15:02:33 on dist-test-slave-qnz0"
server_key: "484f0d2ac2d96446233db8efba2352d0"
server_key_iv: "9a052fe5d225911c9142c4d059ace71d"
server_key_version: "encryptionkey@0"
I20260928 15:02:33.098812 26992 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:33.100274 27007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:33.100734 26992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:33.100916 26992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "880b3a9a93ac43e59c6c7385fc01844a"
format_stamp: "Formatted at 2026-09-28 15:02:33 on dist-test-slave-qnz0"
server_key: "484f0d2ac2d96446233db8efba2352d0"
server_key_iv: "9a052fe5d225911c9142c4d059ace71d"
server_key_version: "encryptionkey@0"
I20260928 15:02:33.101032 26992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:33.115125 26992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:33.118468 26992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:33.118706 26992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:33.119005 26992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:33.119398 26992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:33.119467 26992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:33.119513 26992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:33.119549 26992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:33.129699 26992 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:33873
I20260928 15:02:33.129796 27120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:33873 every 8 connection(s)
I20260928 15:02:33.130263 26992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:33.133455 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 26992
I20260928 15:02:33.133569 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:33.133744 26092 external_mini_cluster.cc:1428] Setting key 62652700e8f34e6c091792c5900978fa
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[26617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:33.147883 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.131496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:53527 (local address 127.25.123.62:35731)
0928 15:02:33.131552 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.131555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.131673 (+ 118us) server_negotiation.cc:407] Connection header received
0928 15:02:33.132961 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.132962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.132998 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.133056 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.134123 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.135161 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.137828 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.138055 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.138155 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.143508 (+ 5353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.143510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.143511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.143522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.145778 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.146502 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.146503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.146504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.146581 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.147021 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.147022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.147022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.147154 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.147209 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.147707 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.147811 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 15:02:33.148103 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.131304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35731 (local address 127.25.123.3:53527)
0928 15:02:33.131480 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.131527 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.132749 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.133143 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.133147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.133692 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:33.133992 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.133999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.135307 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.135308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.137690 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.137697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.138171 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.138172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.138430 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.139828 (+ 1398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:33.139845 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:33.143295 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:33.145921 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:33.145938 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:33.145942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:33.146311 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:33.146675 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:33.146676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:33.146676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:33.146902 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:33.147281 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:33.147288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:33.147459 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.147582 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.147843 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260928 15:02:33.149128 27121 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35731
I20260928 15:02:33.149300 27121 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:33.149673 27121 heartbeater.cc:511] Master 127.25.123.62:35731 requested a full tablet report, sending...
I20260928 15:02:33.150585 26662 ts_manager.cc:194] Registered new tserver with Master: 880b3a9a93ac43e59c6c7385fc01844a (127.25.123.3:33873)
I20260928 15:02:33.151324 26662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:53527
I20260928 15:02:33.158422 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:33.169104 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.160732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.25.123.62:35731)
0928 15:02:33.160905 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.160907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.161015 (+ 108us) server_negotiation.cc:407] Connection header received
0928 15:02:33.161430 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.161431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.161451 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.161505 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.162170 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.162685 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.163446 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.163683 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.163771 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.165170 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.165172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.165172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.165180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.167584 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.168050 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.168052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.168052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.168120 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.168329 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.168330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.168330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.168596 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.168711 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.168872 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.168983 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 15:02:33.172432 26662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:33.174400 26662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:33.182494 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.177811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.25.123.3:33873)
0928 15:02:33.178796 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.178800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.178809 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:33.178844 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.178846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.178862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.178953 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.179062 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.180126 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.181401 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.181997 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.182100 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.182169 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.182354 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.182400 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":115,"threads_started":1}
I20260928 15:02:33.182505 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.177687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:33873 (local address 127.0.0.1:49152)
0928 15:02:33.177889 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.177934 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.178005 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.178960 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.178960 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.178962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.178996 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.178998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.180265 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.180266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.181303 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.181306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.182103 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.182104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.182190 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.182205 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.182334 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.182400 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":40,"thread_start_us":146,"threads_started":1}
I20260928 15:02:33.183293 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.178219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:45567 (local address 127.0.0.1:56992)
0928 15:02:33.178348 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.178357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.178403 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.178884 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.178884 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.178887 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.178963 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.178965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.179861 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.179862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.181147 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.181149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.182728 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.182729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.183099 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.183104 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.183154 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.183223 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260928 15:02:33.183496 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.178111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.25.123.1:39587)
0928 15:02:33.178371 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.178377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.178390 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:02:33.178456 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.178458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.178478 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.178571 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.178972 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.180205 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.181658 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.182883 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.182955 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.183015 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.183333 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.183397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":61,"threads_started":1}
I20260928 15:02:33.183293 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.178008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:39587 (local address 127.0.0.1:48910)
0928 15:02:33.178139 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.178150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.178211 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.178775 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.178778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.178781 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.178873 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.178876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.180315 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.180316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.181463 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.181465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.183050 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.183051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.183112 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.183115 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.183162 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.183204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":32,"thread_start_us":86,"threads_started":1}
I20260928 15:02:33.183651 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.178466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.25.123.2:45567)
0928 15:02:33.178652 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.178656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.178667 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:02:33.178707 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.178709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.178725 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.178800 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.179045 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.179753 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.181245 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.182293 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.182591 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.182673 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.183486 (+ 813us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.183556 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260928 15:02:33.183640 27055 tablet_service.cc:1467] Processing CreateTablet for tablet 82d5ba02b621482392433cdac2937bae (DEFAULT_TABLE table=test-table [id=7c9cd408419e4238b629612b4e3cfc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.184103 27055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d5ba02b621482392433cdac2937bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.184479 26916 tablet_service.cc:1467] Processing CreateTablet for tablet 82d5ba02b621482392433cdac2937bae (DEFAULT_TABLE table=test-table [id=7c9cd408419e4238b629612b4e3cfc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.184808 26916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d5ba02b621482392433cdac2937bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.184751 26787 tablet_service.cc:1467] Processing CreateTablet for tablet 82d5ba02b621482392433cdac2937bae (DEFAULT_TABLE table=test-table [id=7c9cd408419e4238b629612b4e3cfc0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:33.185006 26787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d5ba02b621482392433cdac2937bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:33.187871 27138 tablet_bootstrap.cc:492] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1: Bootstrap starting.
I20260928 15:02:33.188148 27139 tablet_bootstrap.cc:492] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a: Bootstrap starting.
I20260928 15:02:33.188198 27140 tablet_bootstrap.cc:492] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39: Bootstrap starting.
I20260928 15:02:33.188853 27139 tablet_bootstrap.cc:654] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.189347 27140 tablet_bootstrap.cc:654] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.189936 27139 log.cc:824] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:33.190007 27138 tablet_bootstrap.cc:654] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:33.190254 27140 log.cc:824] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:33.190780 27138 log.cc:824] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:33.191107 27139 tablet_bootstrap.cc:492] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a: No bootstrap required, opened a new log
I20260928 15:02:33.191210 27139 ts_tablet_manager.cc:1397] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:02:33.191524 27140 tablet_bootstrap.cc:492] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39: No bootstrap required, opened a new log
I20260928 15:02:33.191614 27140 ts_tablet_manager.cc:1397] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:33.192901 27138 tablet_bootstrap.cc:492] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1: No bootstrap required, opened a new log
I20260928 15:02:33.192991 27138 ts_tablet_manager.cc:1397] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 15:02:33.193635 27139 raft_consensus.cc:348] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.193838 27139 raft_consensus.cc:374] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.193904 27139 raft_consensus.cc:729] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880b3a9a93ac43e59c6c7385fc01844a, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.194082 27139 consensus_queue.cc:260] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.194192 27140 raft_consensus.cc:348] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.194465 27139 ts_tablet_manager.cc:1428] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:33.194528 27121 heartbeater.cc:503] Master 127.25.123.62:35731 was elected leader, sending a full tablet report...
I20260928 15:02:33.194634 27140 raft_consensus.cc:374] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.194744 27140 raft_consensus.cc:729] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2da5821dc440f88ed5611fe4958e39, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.195148 27140 consensus_queue.cc:260] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.195574 26987 heartbeater.cc:503] Master 127.25.123.62:35731 was elected leader, sending a full tablet report...
I20260928 15:02:33.195912 27140 ts_tablet_manager.cc:1428] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:02:33.196547 27138 raft_consensus.cc:348] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.196774 27138 raft_consensus.cc:374] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:33.196849 27138 raft_consensus.cc:729] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29a79de06e142a9aa5c61eefec379b1, State: Initialized, Role: FOLLOWER
I20260928 15:02:33.196976 27138 consensus_queue.cc:260] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.197242 27138 ts_tablet_manager.cc:1428] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260928 15:02:33.197446 26853 heartbeater.cc:503] Master 127.25.123.62:35731 was elected leader, sending a full tablet report...
W20260928 15:02:33.381102 27122 tablet.cc:2369] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:33.392557 26988 tablet.cc:2369] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:33.428586 26854 tablet.cc:2369] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:33.447451 27145 raft_consensus.cc:482] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:33.447640 27145 raft_consensus.cc:504] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.448348 27145 leader_election.cc:288] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 880b3a9a93ac43e59c6c7385fc01844a (127.25.123.3:33873), d29a79de06e142a9aa5c61eefec379b1 (127.25.123.1:39587)
I20260928 15:02:33.454814 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.449100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:54457 (local address 127.25.123.1:39587)
0928 15:02:33.449147 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.449150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.449162 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:02:33.449561 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.449562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.449576 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.449643 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.450356 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.451158 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.453826 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.454356 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.454443 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.454453 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.454668 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.454723 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:02:33.454907 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.448714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:39587 (local address 127.25.123.2:54457)
0928 15:02:33.448871 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.448983 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.449164 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.449662 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.449664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.449666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.450142 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.450153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.451299 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.451300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.453657 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.453664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.454489 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.454490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.454555 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.454559 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.454660 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.454794 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":243,"thread_start_us":128,"threads_started":1}
I20260928 15:02:33.455286 26807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d5ba02b621482392433cdac2937bae" candidate_uuid: "ce2da5821dc440f88ed5611fe4958e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29a79de06e142a9aa5c61eefec379b1" is_pre_election: true
I20260928 15:02:33.455446 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.448519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:33873 (local address 127.25.123.2:58437)
0928 15:02:33.448678 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:33.448935 (+ 257us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:33.449009 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:33.449700 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:33.449702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:33.449704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:33.449857 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.449860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.451052 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.451054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.453678 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:33.453683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.455091 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.455092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:33.455235 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.455239 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:33.455312 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 15:02:33.455354 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":180,"thread_start_us":113,"threads_started":1}
I20260928 15:02:33.455472 26807 raft_consensus.cc:2454] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce2da5821dc440f88ed5611fe4958e39 in term 0.
I20260928 15:02:33.455482 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.449178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:58437 (local address 127.25.123.3:33873)
0928 15:02:33.449226 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.449229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.449237 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:33.449269 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.449271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.449281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.449476 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:33.450145 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.450960 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.454494 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.454989 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.455082 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.455091 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.455317 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.455369 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:02:33.455745 26874 leader_election.cc:302] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce2da5821dc440f88ed5611fe4958e39, d29a79de06e142a9aa5c61eefec379b1; no voters:
I20260928 15:02:33.455907 27145 raft_consensus.cc:2781] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:33.455977 27145 raft_consensus.cc:482] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:33.456017 27145 raft_consensus.cc:3037] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.456568 27074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d5ba02b621482392433cdac2937bae" candidate_uuid: "ce2da5821dc440f88ed5611fe4958e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880b3a9a93ac43e59c6c7385fc01844a" is_pre_election: true
I20260928 15:02:33.456738 27074 raft_consensus.cc:2454] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce2da5821dc440f88ed5611fe4958e39 in term 0.
I20260928 15:02:33.457271 27145 raft_consensus.cc:504] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.457481 27145 leader_election.cc:288] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [CANDIDATE]: Term 1 election: Requested vote from peers 880b3a9a93ac43e59c6c7385fc01844a (127.25.123.3:33873), d29a79de06e142a9aa5c61eefec379b1 (127.25.123.1:39587)
I20260928 15:02:33.457659 27074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d5ba02b621482392433cdac2937bae" candidate_uuid: "ce2da5821dc440f88ed5611fe4958e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880b3a9a93ac43e59c6c7385fc01844a"
I20260928 15:02:33.457712 26807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d5ba02b621482392433cdac2937bae" candidate_uuid: "ce2da5821dc440f88ed5611fe4958e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29a79de06e142a9aa5c61eefec379b1"
I20260928 15:02:33.457775 27074 raft_consensus.cc:3037] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.457803 26807 raft_consensus.cc:3037] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:33.458977 27074 raft_consensus.cc:2454] T 82d5ba02b621482392433cdac2937bae P 880b3a9a93ac43e59c6c7385fc01844a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce2da5821dc440f88ed5611fe4958e39 in term 1.
I20260928 15:02:33.459016 26807 raft_consensus.cc:2454] T 82d5ba02b621482392433cdac2937bae P d29a79de06e142a9aa5c61eefec379b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce2da5821dc440f88ed5611fe4958e39 in term 1.
I20260928 15:02:33.459204 26874 leader_election.cc:302] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce2da5821dc440f88ed5611fe4958e39, d29a79de06e142a9aa5c61eefec379b1; no voters:
I20260928 15:02:33.459447 27145 raft_consensus.cc:2781] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:33.459662 27145 raft_consensus.cc:686] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [term 1 LEADER]: Becoming Leader. State: Replica: ce2da5821dc440f88ed5611fe4958e39, State: Running, Role: LEADER
I20260928 15:02:33.459857 27145 consensus_queue.cc:237] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } }
I20260928 15:02:33.461012 26662 catalog_manager.cc:5495] T 82d5ba02b621482392433cdac2937bae P ce2da5821dc440f88ed5611fe4958e39 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce2da5821dc440f88ed5611fe4958e39 (127.25.123.2). New cstate: current_term: 1 leader_uuid: "ce2da5821dc440f88ed5611fe4958e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880b3a9a93ac43e59c6c7385fc01844a" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d29a79de06e142a9aa5c61eefec379b1" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce2da5821dc440f88ed5611fe4958e39" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 45567 } health_report { overall_health: HEALTHY } } }
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[26617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:02:33.478763 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.468866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.25.123.1:39587)
0928 15:02:33.468941 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.468944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.468953 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:33.468977 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.468978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.468988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.469043 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.469906 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.470404 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.471315 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.471511 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.471576 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.475001 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.475018 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.475018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.475028 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.477213 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.477701 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.477705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.477706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.477870 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.478114 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.478115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.478116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.478293 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.478480 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.478523 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.478626 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[26617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[26617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:02:33.523154 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.512121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.25.123.1:39587)
0928 15:02:33.512205 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.512208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.512378 (+ 170us) server_negotiation.cc:407] Connection header received
0928 15:02:33.512547 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.512548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.512562 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.512655 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.513491 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.514045 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.514800 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.515012 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.515096 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.519887 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.519889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.519890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.519909 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.521658 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.522133 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.522135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.522135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.522203 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.522457 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.522459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.522459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.522585 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.522709 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.522935 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.523033 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[26617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[26617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:02:33.563503 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:33.549513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.25.123.1:39587)
0928 15:02:33.549586 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:33.549588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:33.549736 (+ 148us) server_negotiation.cc:407] Connection header received
0928 15:02:33.549861 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:33.549863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:33.549875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:33.550280 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:33.551537 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.552131 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.554245 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:33.554495 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:33.554761 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:33.559380 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:33.559383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:33.559383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:33.559394 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:33.561428 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.562341 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.562343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.562343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.562426 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:33.562755 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:33.562755 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:33.562756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:33.562892 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:33.562980 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:33.563323 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 15:02:33.563422 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 15:02:33.564236 26767 tablet_service.cc:2908] Rejecting scan request for tablet 82d5ba02b621482392433cdac2937bae: Uninitialized: clean time has not yet been initialized
I20260928 15:02:33.566344 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26724
I20260928 15:02:33.575970 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26858
I20260928 15:02:33.582706 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26992
I20260928 15:02:33.588195 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 26630
2026-09-28T15:02:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2832 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[27163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[27163](info): set up 2 sockets
Sep 28 15:02:33 dist-test-slave-qnz0 krb5kdc[27163](info): commencing operation
krb5kdc: starting...
W20260928 15:02:34.969365 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.354s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:35 dist-test-slave-qnz0 krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:35.153292 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35699
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:35.300904 27176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:35.301450 27176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:35.301553 27176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:35.303684 27176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:35.303790 27176 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:02:35.303815 27176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:35.303841 27176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:35.303875 27176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:35.303903 27176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:35.306725 27176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:35699
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:35.307541 27176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:35.307857 27176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:35.315193 27182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:35.315481 27176 server_base.cc:1034] running on GCE node
W20260928 15:02:35.315332 27184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:35.315303 27181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:35.316560 27176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:35.316958 27176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:35.318189 27176 hybrid_clock.cc:648] HybridClock initialized: now 1790607755318101 us; error 106 us; skew 500 ppm
Sep 28 15:02:35 dist-test-slave-qnz0 krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:35.322136 27176 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:35.322782 27176 webserver.cc:460] Webserver started at http://127.25.123.62:42967/ using document root <none> and password file <none>
I20260928 15:02:35.323240 27176 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:35.323324 27176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:35.323470 27176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:35.325317 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "266be9d500b141828df9249ace196d5e"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "613458a3a9ba752d8c6f26bf59070aa5"
server_key_iv: "42c366479857e8459b65c296818610f0"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.325858 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "266be9d500b141828df9249ace196d5e"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "613458a3a9ba752d8c6f26bf59070aa5"
server_key_iv: "42c366479857e8459b65c296818610f0"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.328277 27176 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:02:35.330447 27191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:35.330853 27176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:02:35.331005 27176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "266be9d500b141828df9249ace196d5e"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "613458a3a9ba752d8c6f26bf59070aa5"
server_key_iv: "42c366479857e8459b65c296818610f0"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.331144 27176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:35.346487 27176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:35.349448 27176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:35.349645 27176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:35.357451 27176 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:35699
I20260928 15:02:35.357553 27253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:35699 every 8 connection(s)
I20260928 15:02:35.357901 27176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:35.358621 27254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:35.361923 27254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e: Bootstrap starting.
I20260928 15:02:35.362313 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 27176
I20260928 15:02:35.362514 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:35.362746 26092 external_mini_cluster.cc:1428] Setting key 4b1e728983905f07a6450c95732d208f
I20260928 15:02:35.362751 27254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:35.363124 27254 log.cc:824] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:35.364099 27254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e: No bootstrap required, opened a new log
I20260928 15:02:35.366184 27254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266be9d500b141828df9249ace196d5e" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35699 } }
I20260928 15:02:35.366350 27254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:35.366375 27254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266be9d500b141828df9249ace196d5e, State: Initialized, Role: FOLLOWER
I20260928 15:02:35.366516 27254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266be9d500b141828df9249ace196d5e" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35699 } }
I20260928 15:02:35.366595 27254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:35.366627 27254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:35.366683 27254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:35.368022 27254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266be9d500b141828df9249ace196d5e" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35699 } }
I20260928 15:02:35.368377 27254 leader_election.cc:302] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 266be9d500b141828df9249ace196d5e; no voters:
I20260928 15:02:35.368803 27254 leader_election.cc:288] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:35.368839 27259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:35.369087 27259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [term 1 LEADER]: Becoming Leader. State: Replica: 266be9d500b141828df9249ace196d5e, State: Running, Role: LEADER
I20260928 15:02:35.369212 27259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266be9d500b141828df9249ace196d5e" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35699 } }
I20260928 15:02:35.369624 27259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "266be9d500b141828df9249ace196d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266be9d500b141828df9249ace196d5e" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35699 } } }
I20260928 15:02:35.369642 27260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 266be9d500b141828df9249ace196d5e. Latest consensus state: current_term: 1 leader_uuid: "266be9d500b141828df9249ace196d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266be9d500b141828df9249ace196d5e" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35699 } } }
I20260928 15:02:35.370141 27260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:35.369966 27259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:35.369971 27254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:35.370765 27263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:35.371476 27263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:02:35 dist-test-slave-qnz0 krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:35.374030 27263 catalog_manager.cc:1347] Generated new cluster ID: c29e70e2351245cb9f6facb39c141760
I20260928 15:02:35.374094 27263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:35.383687 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.363465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.25.123.62:35699)
0928 15:02:35.363624 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:35.363635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:35.363671 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:02:35.364469 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:35.364478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:35.364536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:35.364703 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:35.365485 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.367036 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.368233 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.368938 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.369440 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.373600 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:35.373605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:35.373606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:35.373620 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:35.381012 (+ 7392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.381679 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.381685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.381685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.381834 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.382327 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.382331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.382331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.382784 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:35.382984 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:35.383120 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 15:02:35.383420 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20260928 15:02:35.390426 27263 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:35.391129 27263 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:35.397495 27263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e: Generated new TSK 0
I20260928 15:02:35.397785 27263 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:35.452991 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35699
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:35.573865 27280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:35.574707 27280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:35.574816 27280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:35.576581 27280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:35.576640 27280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:35.576668 27280 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:35.576689 27280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:35.579916 27280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:35.580298 27280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:35.580674 27280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:35.589375 27288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:35.589563 27285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:35.590448 27286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:35.590391 27280 server_base.cc:1034] running on GCE node
I20260928 15:02:35.592301 27280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:35.592971 27280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:35.597550 27280 hybrid_clock.cc:648] HybridClock initialized: now 1790607755597978 us; error 630 us; skew 500 ppm
Sep 28 15:02:35 dist-test-slave-qnz0 krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:35.610010 27280 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:35.610611 27280 webserver.cc:460] Webserver started at http://127.25.123.1:34927/ using document root <none> and password file <none>
I20260928 15:02:35.610910 27280 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:35.610993 27280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:35.611510 27280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:35.614034 27280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "cec60188e274420cb5fe8dd5c12bdbaf"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "d52473e8e585625c15d9bbe2d08fc5f4"
server_key_iv: "bed6f761dfd11c34742f030a32250049"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.614619 27280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "cec60188e274420cb5fe8dd5c12bdbaf"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "d52473e8e585625c15d9bbe2d08fc5f4"
server_key_iv: "bed6f761dfd11c34742f030a32250049"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.616863 27280 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 15:02:35.618620 27295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:35.619123 27280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:35.619501 27280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "cec60188e274420cb5fe8dd5c12bdbaf"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "d52473e8e585625c15d9bbe2d08fc5f4"
server_key_iv: "bed6f761dfd11c34742f030a32250049"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.619616 27280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:35.666543 27280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:35.672248 27280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:35.672490 27280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:35.673065 27280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:35.673134 27280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:35.673179 27280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:35.673205 27280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:35.686014 27280 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:44229
I20260928 15:02:35.686069 27410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:44229 every 8 connection(s)
I20260928 15:02:35.687001 27280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
Sep 28 15:02:35 dist-test-slave-qnz0 krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:35.693881 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 27280
I20260928 15:02:35.693996 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:35.694185 26092 external_mini_cluster.cc:1428] Setting key ff0e59c2cfaf48763ff391c8faa5efde
I20260928 15:02:35.700728 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.677388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:58583 (local address 127.25.123.62:35699)
0928 15:02:35.677445 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:35.677447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:35.677466 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:02:35.679019 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:35.679021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:35.679036 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:35.679113 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:35.683972 (+ 4859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.685014 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.686626 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.686997 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.687361 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.691787 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:35.691791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:35.691791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:35.691802 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:35.694123 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.698668 (+ 4545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.698671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.698671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.698827 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.699551 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.699553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.699554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.699812 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:35.699925 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:35.700437 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 15:02:35.700559 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:02:35.701004 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.677025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35699 (local address 127.25.123.1:58583)
0928 15:02:35.677205 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:35.677252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:35.678254 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:35.681034 (+ 2780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:35.681041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:35.683205 (+ 2164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:35.683735 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.683748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.685164 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.685165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.686497 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.686501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.687500 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.687502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.687660 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.688737 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:35.688749 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:35.691631 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:35.694374 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.694391 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.694395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.698431 (+ 4036us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.698915 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.698916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.698917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.699174 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.699921 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:35.699938 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:35.700214 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 15:02:35.700369 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 15:02:35.700713 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 15:02:35.701902 27411 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35699
I20260928 15:02:35.702018 27411 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:35.702263 27411 heartbeater.cc:511] Master 127.25.123.62:35699 requested a full tablet report, sending...
I20260928 15:02:35.702977 27207 ts_manager.cc:194] Registered new tserver with Master: cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1:44229)
I20260928 15:02:35.703926 27207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:58583
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
I20260928 15:02:35.717454 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.704695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:60855 (local address 127.25.123.62:35699)
0928 15:02:35.704742 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:35.704745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:35.706530 (+ 1785us) server_negotiation.cc:407] Connection header received
0928 15:02:35.706660 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:35.706661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:35.706674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:35.706727 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:35.707114 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.708142 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.709684 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.709965 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.710043 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.711623 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:35.711628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:35.711628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:35.711641 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:35.714741 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.715358 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.715362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.715362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.715500 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.715999 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.716000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.716001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.716251 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:35.716347 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:35.717235 (+ 888us) server_negotiation.cc:299] Negotiation successful
0928 15:02:35.717340 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:02:35.717470 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.704780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35699 (local address 127.25.123.1:60855)
0928 15:02:35.706420 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:35.706435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:35.706564 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:35.706732 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:35.706733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:35.706951 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:35.707046 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.707049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.708254 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.708256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.709531 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.709535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.710044 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.710045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.710137 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.710665 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:35.710687 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:35.711375 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:35.714882 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.714883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.714884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.715161 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.715717 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.715719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.715719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.715870 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.716450 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:35.716451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:35.716740 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 15:02:35.717170 (+ 430us) client_negotiation.cc:240] Negotiation successful
0928 15:02:35.717343 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1598,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:35.747371 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35699
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:35.864473 27420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:35.865065 27420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:35.865171 27420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:35.866681 27420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:35.866740 27420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:35.866765 27420 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:35.866803 27420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:35.868630 27420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:35.868948 27420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:35.869220 27420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:35.879902 27426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:35.880466 27425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:35.880530 27428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:35.880661 27420 server_base.cc:1034] running on GCE node
I20260928 15:02:35.881108 27420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:35.881431 27420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:35.882663 27420 hybrid_clock.cc:648] HybridClock initialized: now 1790607755882633 us; error 43 us; skew 500 ppm
Sep 28 15:02:35 dist-test-slave-qnz0 krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:35.887423 27420 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:35.888299 27420 webserver.cc:460] Webserver started at http://127.25.123.2:46271/ using document root <none> and password file <none>
I20260928 15:02:35.888906 27420 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:35.889243 27420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:35.890302 27420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:35.891813 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "8d2986836c1346d19df6c41c48ef55ef"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "3543e735c606ec5db44fd4732af7b3d7"
server_key_iv: "8daa501aa0aafb1e36af0514e8b6ce95"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.892628 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "8d2986836c1346d19df6c41c48ef55ef"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "3543e735c606ec5db44fd4732af7b3d7"
server_key_iv: "8daa501aa0aafb1e36af0514e8b6ce95"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.895486 27420 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:35.897205 27435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:35.897698 27420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:35.897863 27420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "8d2986836c1346d19df6c41c48ef55ef"
format_stamp: "Formatted at 2026-09-28 15:02:35 on dist-test-slave-qnz0"
server_key: "3543e735c606ec5db44fd4732af7b3d7"
server_key_iv: "8daa501aa0aafb1e36af0514e8b6ce95"
server_key_version: "encryptionkey@0"
I20260928 15:02:35.898082 27420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:35.919806 27420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:35.924185 27420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:35.924546 27420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:35.925441 27420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:35.925732 27420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:35.925936 27420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:35.925964 27420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:35 dist-test-slave-qnz0 krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:35.939994 27420 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:41503
I20260928 15:02:35.940153 27550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:41503 every 8 connection(s)
I20260928 15:02:35.940780 27420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:35.942004 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.926082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35699 (local address 127.25.123.2:52731)
0928 15:02:35.926275 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:35.926320 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:35.927272 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:35.927594 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:35.927598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:35.927922 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:35.928517 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.928524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.929292 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.929293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.930981 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.930986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.931509 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.931510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.931710 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.932658 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:35.932769 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:35.936298 (+ 3529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:35.939224 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.939232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.939236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.939705 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.940060 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.940061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.940061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.940222 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.940774 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:35.940784 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:35.941037 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 15:02:35.941227 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 15:02:35.941671 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
I20260928 15:02:35.942452 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.926081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:52731 (local address 127.25.123.62:35699)
0928 15:02:35.926139 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:35.926142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:35.926400 (+ 258us) server_negotiation.cc:407] Connection header received
0928 15:02:35.927440 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:35.927441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:35.927457 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:35.927511 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:35.928620 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.929165 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.931101 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.931398 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.931487 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.936553 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:35.936556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:35.936557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:35.936569 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:35.939105 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.939877 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.939880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.939880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.939976 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.940438 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.940441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.940442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.940636 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:35.940724 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:35.942234 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0928 15:02:35.942373 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:02:35.942919 27551 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35699
I20260928 15:02:35.943029 27551 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:35.943270 27551 heartbeater.cc:511] Master 127.25.123.62:35699 requested a full tablet report, sending...
I20260928 15:02:35.943706 27207 ts_manager.cc:194] Registered new tserver with Master: 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2:41503)
I20260928 15:02:35.944272 27207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:52731
I20260928 15:02:35.946257 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 27420
I20260928 15:02:35.946485 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:35.947074 26092 external_mini_cluster.cc:1428] Setting key 1f69cd1fec2cc6779e65fe5900dd99fd
I20260928 15:02:35.960776 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.944187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:40499 (local address 127.25.123.62:35699)
0928 15:02:35.944264 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:35.944267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:35.944579 (+ 312us) server_negotiation.cc:407] Connection header received
0928 15:02:35.944778 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:35.944779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:35.944793 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:35.944864 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:35.945998 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.947263 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.948802 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.949016 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.949111 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.951835 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:35.951838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:35.951839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:35.951851 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:35.957760 (+ 5909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.958475 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.958477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.958478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.958549 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:35.959286 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:35.959287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:35.959287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:35.959759 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:35.959854 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:35.960136 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 15:02:35.960665 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:02:35.961401 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:35.944305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35699 (local address 127.25.123.2:40499)
0928 15:02:35.944513 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:35.944526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:35.944609 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:35.944883 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:35.944884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:35.945159 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:35.945230 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.945233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.947459 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.947461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.948497 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:35.948504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:35.949229 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:35.949231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:35.949392 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:35.950720 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:35.950728 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:35.951628 (+ 900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:35.957967 (+ 6339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.957970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.957971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.958300 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.958643 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:35.958644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:35.958645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:35.958730 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:35.959870 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:35.959872 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:35.959950 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 15:02:35.960058 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:02:35.960228 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:36.029580 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35699
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:36.145371 27560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:36.145758 27560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:36.145913 27560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:36.147917 27560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:36.148049 27560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:36.148092 27560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:36.148175 27560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:36.150974 27560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:36.151330 27560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:36.151623 27560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:36.158116 27565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:36.158109 27568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:36.158108 27566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:36.158587 27560 server_base.cc:1034] running on GCE node
I20260928 15:02:36.158888 27560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:36.159348 27560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:36.160569 27560 hybrid_clock.cc:648] HybridClock initialized: now 1790607756160500 us; error 90 us; skew 500 ppm
Sep 28 15:02:36 dist-test-slave-qnz0 krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:36.165349 27560 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:36.166136 27560 webserver.cc:460] Webserver started at http://127.25.123.3:43309/ using document root <none> and password file <none>
I20260928 15:02:36.166484 27560 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:36.166563 27560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:36.166728 27560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:36.168002 27560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "e39e368b27584db89b0774254f47fe47"
format_stamp: "Formatted at 2026-09-28 15:02:36 on dist-test-slave-qnz0"
server_key: "5049f2e51a1f770a3798d46a4b99b841"
server_key_iv: "4a11334ea651250cbeef525bfb8f28d6"
server_key_version: "encryptionkey@0"
I20260928 15:02:36.168455 27560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "e39e368b27584db89b0774254f47fe47"
format_stamp: "Formatted at 2026-09-28 15:02:36 on dist-test-slave-qnz0"
server_key: "5049f2e51a1f770a3798d46a4b99b841"
server_key_iv: "4a11334ea651250cbeef525bfb8f28d6"
server_key_version: "encryptionkey@0"
I20260928 15:02:36.170032 27560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:36.171412 27575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:36.171743 27560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:02:36.171869 27560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "e39e368b27584db89b0774254f47fe47"
format_stamp: "Formatted at 2026-09-28 15:02:36 on dist-test-slave-qnz0"
server_key: "5049f2e51a1f770a3798d46a4b99b841"
server_key_iv: "4a11334ea651250cbeef525bfb8f28d6"
server_key_version: "encryptionkey@0"
I20260928 15:02:36.171965 27560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:36.213027 27560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:36.216554 27560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:36.217097 27560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:36.218470 27560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:36.218607 27560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:36.218640 27560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:36.218654 27560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:36 dist-test-slave-qnz0 krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:36.231050 27560 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:35877
I20260928 15:02:36.231550 27560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:36.232532 27690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:35877 every 8 connection(s)
I20260928 15:02:36.233067 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.219296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:57291 (local address 127.25.123.62:35699)
0928 15:02:36.219437 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.219441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.219789 (+ 348us) server_negotiation.cc:407] Connection header received
0928 15:02:36.220471 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.220473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.220488 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.220601 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:36.221344 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.222383 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.224835 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.225108 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.225229 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.228457 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:36.228461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:36.228462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:36.228475 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:36.230756 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.231587 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.231590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.231591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.231681 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.232057 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.232057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.232058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.232266 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:36.232360 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.232881 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.232984 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 15:02:36.233280 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.219257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35699 (local address 127.25.123.3:57291)
0928 15:02:36.219592 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.219646 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.220282 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.220702 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.220706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.221005 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:36.221230 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.221235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.222515 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.222515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.224695 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.224702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.225246 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.225247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.225419 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.226145 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:36.226159 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:36.228186 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:36.230976 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:36.230993 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:36.230997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:36.231371 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:36.231778 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:36.231779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:36.231779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:36.231963 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:36.232446 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:36.232452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:36.232676 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.232832 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.233013 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":227,"threads_started":1}
I20260928 15:02:36.234134 27691 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35699
I20260928 15:02:36.234305 27691 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:36.234612 27691 heartbeater.cc:511] Master 127.25.123.62:35699 requested a full tablet report, sending...
I20260928 15:02:36.235471 27208 ts_manager.cc:194] Registered new tserver with Master: e39e368b27584db89b0774254f47fe47 (127.25.123.3:35877)
I20260928 15:02:36.236131 27208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:57291
I20260928 15:02:36.238948 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 27560
I20260928 15:02:36.239100 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:36.239323 26092 external_mini_cluster.cc:1428] Setting key 7a63d8cf30355d201db2fe4061b3926b
I20260928 15:02:36.241076 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:36.244232 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.235992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35699 (local address 127.25.123.3:43885)
0928 15:02:36.236250 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.236494 (+ 244us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.236589 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.236953 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.236954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.237166 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:36.237251 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.237253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.238122 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.238123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.238779 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.238782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.239283 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.239284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.239366 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.240200 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:36.240208 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:36.240802 (+ 594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:36.242885 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:36.242886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:36.242887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:36.243087 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:36.243325 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:36.243326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:36.243326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:36.243436 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:36.243746 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:36.243747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:36.243825 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.243906 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.244036 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":136,"threads_started":1}
I20260928 15:02:36.244318 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.236021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:43885 (local address 127.25.123.62:35699)
0928 15:02:36.236346 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.236349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.236564 (+ 215us) server_negotiation.cc:407] Connection header received
0928 15:02:36.236801 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.236802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.236817 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.236887 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:36.237467 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.237996 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.238921 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.239205 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.239274 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.240906 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:36.240908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:36.240908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:36.240918 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:36.242792 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.243168 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.243170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.243170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.243232 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.243500 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.243501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.243501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.243642 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:36.243727 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.243936 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.244025 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
Sep 28 15:02:36 dist-test-slave-qnz0 krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:36 dist-test-slave-qnz0 krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:36.272382 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.262490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.25.123.62:35699)
0928 15:02:36.262577 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.262581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.262590 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:36.262627 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.262628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.262637 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.262718 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:36.263415 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.263947 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.265274 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.265482 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.265579 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.268759 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:36.268762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:36.268763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:36.268780 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:36.270816 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.271274 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.271276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.271277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.271340 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.271738 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.271739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.271740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.271887 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:36.272009 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.272173 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.272267 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:02:36.279860 27208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:36.283715 27208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:36.294453 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.289588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:41503 (local address 127.0.0.1:53068)
0928 15:02:36.289974 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.290018 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.290095 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.290393 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.290396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.290399 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.290481 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.290483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.291928 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.291930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.293212 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.293214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.294077 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.294078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.294172 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.294188 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.294269 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.294333 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":128,"threads_started":1}
I20260928 15:02:36.294462 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.289749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.25.123.2:41503)
0928 15:02:36.290236 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.290240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.290248 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:36.290280 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.290282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.290295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.290387 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.290575 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.291777 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.293282 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.293974 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.294077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.294157 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.294297 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.294343 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":134,"threads_started":1}
I20260928 15:02:36.296761 27485 tablet_service.cc:1467] Processing CreateTablet for tablet daf72486526c4c65b04d85e514620eec (DEFAULT_TABLE table=test-table [id=c7ca4e88fa104c59a8d526a0e4c0d73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:36.297329 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf72486526c4c65b04d85e514620eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.300446 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.289964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.25.123.1:44229)
0928 15:02:36.290117 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.290123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.290156 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:02:36.290194 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.290197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.290215 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.293107 (+ 2892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.293211 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.296965 (+ 3754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.298673 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.299794 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.299997 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.300079 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.300175 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.300256 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260928 15:02:36.300727 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.289844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44229 (local address 127.0.0.1:37592)
0928 15:02:36.289972 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.290018 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.290095 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.293083 (+ 2988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.293084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.293086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.293137 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.293139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.297104 (+ 3965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.297105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.298524 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.298527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.299999 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.300000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.300069 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.300075 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.300147 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.300200 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 15:02:36.301961 27714 tablet_bootstrap.cc:492] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Bootstrap starting.
I20260928 15:02:36.302225 27344 tablet_service.cc:1467] Processing CreateTablet for tablet daf72486526c4c65b04d85e514620eec (DEFAULT_TABLE table=test-table [id=c7ca4e88fa104c59a8d526a0e4c0d73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:36.302713 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf72486526c4c65b04d85e514620eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.303095 27714 tablet_bootstrap.cc:654] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.304505 27714 log.cc:824] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:36.305533 27714 tablet_bootstrap.cc:492] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: No bootstrap required, opened a new log
I20260928 15:02:36.305608 27714 ts_tablet_manager.cc:1397] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 15:02:36.307027 27716 tablet_bootstrap.cc:492] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Bootstrap starting.
I20260928 15:02:36.307273 27714 raft_consensus.cc:348] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.307483 27714 raft_consensus.cc:374] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.307531 27714 raft_consensus.cc:729] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2986836c1346d19df6c41c48ef55ef, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.307689 27714 consensus_queue.cc:260] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.307780 27716 tablet_bootstrap.cc:654] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.307929 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.289503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:35877 (local address 127.0.0.1:36588)
0928 15:02:36.289970 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.290018 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.290095 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.293438 (+ 3343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.293438 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.293441 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.293479 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.293481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.297817 (+ 4336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.297819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.298952 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.298956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.307577 (+ 8621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.307579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.307692 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.307701 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.307743 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.307806 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":84,"threads_started":1}
I20260928 15:02:36.308017 27714 ts_tablet_manager.cc:1428] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:02:36.308243 27716 log.cc:824] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:36.308269 27551 heartbeater.cc:503] Master 127.25.123.62:35699 was elected leader, sending a full tablet report...
I20260928 15:02:36.308375 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.289736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.25.123.3:35877)
0928 15:02:36.293020 (+ 3284us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.293026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.293043 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:02:36.293105 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.293107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.293126 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.293431 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.293569 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.297675 (+ 4106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.299467 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.307431 (+ 7964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.308084 (+ 653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.308174 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.308206 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.308262 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":123,"threads_started":1}
I20260928 15:02:36.309059 27716 tablet_bootstrap.cc:492] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: No bootstrap required, opened a new log
I20260928 15:02:36.309166 27716 ts_tablet_manager.cc:1397] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:36.309139 27625 tablet_service.cc:1467] Processing CreateTablet for tablet daf72486526c4c65b04d85e514620eec (DEFAULT_TABLE table=test-table [id=c7ca4e88fa104c59a8d526a0e4c0d73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:36.309549 27625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf72486526c4c65b04d85e514620eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.313030 27719 tablet_bootstrap.cc:492] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Bootstrap starting.
I20260928 15:02:36.314014 27719 tablet_bootstrap.cc:654] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.314473 27719 log.cc:824] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:36.314319 27716 raft_consensus.cc:348] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.314814 27716 raft_consensus.cc:374] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.314894 27716 raft_consensus.cc:729] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec60188e274420cb5fe8dd5c12bdbaf, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.315055 27716 consensus_queue.cc:260] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.315393 27716 ts_tablet_manager.cc:1428] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 15:02:36.315477 27719 tablet_bootstrap.cc:492] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: No bootstrap required, opened a new log
I20260928 15:02:36.315548 27411 heartbeater.cc:503] Master 127.25.123.62:35699 was elected leader, sending a full tablet report...
I20260928 15:02:36.315574 27719 ts_tablet_manager.cc:1397] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:02:36.317724 27719 raft_consensus.cc:348] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.317934 27719 raft_consensus.cc:374] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.317983 27719 raft_consensus.cc:729] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39e368b27584db89b0774254f47fe47, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.318117 27719 consensus_queue.cc:260] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.318425 27719 ts_tablet_manager.cc:1428] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:36.318567 27691 heartbeater.cc:503] Master 127.25.123.62:35699 was elected leader, sending a full tablet report...
I20260928 15:02:36.328989 27722 raft_consensus.cc:482] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:36.329396 27722 raft_consensus.cc:504] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.329885 27722 leader_election.cc:288] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2:41503), cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1:44229)
I20260928 15:02:36.334901 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.330149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:41503 (local address 127.25.123.3:56919)
0928 15:02:36.330230 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.330246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.330318 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.330512 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.330513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.330516 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.330600 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.330602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.331976 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.331978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.333044 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.333049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.334534 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.334535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.334644 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.334650 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.334742 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.334803 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":45}
I20260928 15:02:36.335450 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.330258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:56919 (local address 127.25.123.2:41503)
0928 15:02:36.330331 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.330333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.330347 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:02:36.330380 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.330381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.330391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.330620 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.330668 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.331810 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.333732 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.334426 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.334504 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.334515 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.334845 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.334923 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:02:36.335597 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.330280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44229 (local address 127.25.123.3:55125)
0928 15:02:36.330509 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.330520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.330587 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.330819 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.330820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.330821 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.330884 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.330885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.332030 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.332032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.334349 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.334353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.335301 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.335302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.335382 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.335385 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.335460 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.335503 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
I20260928 15:02:36.335594 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.330366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:55125 (local address 127.25.123.1:44229)
0928 15:02:36.330432 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.330434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.330620 (+ 186us) server_negotiation.cc:407] Connection header received
0928 15:02:36.330666 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.330667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.330681 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.330753 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.330963 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.331891 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.334461 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.335144 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.335225 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.335255 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.335476 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.335522 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:02:36.335939 27360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf72486526c4c65b04d85e514620eec" candidate_uuid: "e39e368b27584db89b0774254f47fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" is_pre_election: true
I20260928 15:02:36.335983 27505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf72486526c4c65b04d85e514620eec" candidate_uuid: "e39e368b27584db89b0774254f47fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2986836c1346d19df6c41c48ef55ef" is_pre_election: true
I20260928 15:02:36.336128 27360 raft_consensus.cc:2454] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e39e368b27584db89b0774254f47fe47 in term 0.
I20260928 15:02:36.336128 27505 raft_consensus.cc:2454] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e39e368b27584db89b0774254f47fe47 in term 0.
I20260928 15:02:36.336440 27578 leader_election.cc:302] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec60188e274420cb5fe8dd5c12bdbaf, e39e368b27584db89b0774254f47fe47; no voters:
I20260928 15:02:36.336655 27722 raft_consensus.cc:2781] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:36.336755 27722 raft_consensus.cc:482] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:36.336777 27722 raft_consensus.cc:3037] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.337807 27722 raft_consensus.cc:504] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.337946 27722 leader_election.cc:288] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [CANDIDATE]: Term 1 election: Requested vote from peers 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2:41503), cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1:44229)
I20260928 15:02:36.338282 27505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf72486526c4c65b04d85e514620eec" candidate_uuid: "e39e368b27584db89b0774254f47fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2986836c1346d19df6c41c48ef55ef"
I20260928 15:02:36.338387 27505 raft_consensus.cc:3037] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.339190 27505 raft_consensus.cc:2454] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e39e368b27584db89b0774254f47fe47 in term 1.
I20260928 15:02:36.339087 27360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf72486526c4c65b04d85e514620eec" candidate_uuid: "e39e368b27584db89b0774254f47fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec60188e274420cb5fe8dd5c12bdbaf"
I20260928 15:02:36.339421 27360 raft_consensus.cc:3037] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.339449 27578 leader_election.cc:302] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2986836c1346d19df6c41c48ef55ef, e39e368b27584db89b0774254f47fe47; no voters:
I20260928 15:02:36.339982 27722 raft_consensus.cc:2781] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:36.340620 27722 raft_consensus.cc:686] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 1 LEADER]: Becoming Leader. State: Replica: e39e368b27584db89b0774254f47fe47, State: Running, Role: LEADER
I20260928 15:02:36.340830 27722 consensus_queue.cc:237] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.340999 27360 raft_consensus.cc:2454] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e39e368b27584db89b0774254f47fe47 in term 1.
I20260928 15:02:36.342075 27207 catalog_manager.cc:5495] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 reported cstate change: term changed from 0 to 1, leader changed from <none> to e39e368b27584db89b0774254f47fe47 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "e39e368b27584db89b0774254f47fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } health_report { overall_health: UNKNOWN } } }
I20260928 15:02:36.362787 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.358514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.25.123.3:35877)
0928 15:02:36.358610 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.358612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.358620 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:02:36.358657 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.358658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.358678 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.358753 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:36.359259 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.359766 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.361408 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.361797 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.361875 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.362173 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.362327 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.362445 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.362685 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.362715 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:02:36.366699 27360 raft_consensus.cc:1261] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Refusing update from remote peer e39e368b27584db89b0774254f47fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:36.366971 27505 raft_consensus.cc:1261] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Refusing update from remote peer e39e368b27584db89b0774254f47fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:36.367271 27724 consensus_queue.cc:1035] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:36.367511 27722 consensus_queue.cc:1035] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:36.372639 27727 mvcc.cc:205] Tried to move back new op lower bound from 7334329370072711168 to 7334329369972965376. Current Snapshot: MvccSnapshot[applied={T|T < 7334329370072711168}]
I20260928 15:02:36.372742 27728 mvcc.cc:205] Tried to move back new op lower bound from 7334329370072711168 to 7334329369972965376. Current Snapshot: MvccSnapshot[applied={T|T < 7334329370072711168}]
I20260928 15:02:36.373061 27730 mvcc.cc:205] Tried to move back new op lower bound from 7334329370072711168 to 7334329369972965376. Current Snapshot: MvccSnapshot[applied={T|T < 7334329370072711168}]
I20260928 15:02:36.379485 27207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:36.379621 27207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:36.381625 27207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e: Sending DeleteTablet for 3 replicas of tablet daf72486526c4c65b04d85e514620eec
I20260928 15:02:36.382301 27625 tablet_service.cc:1514] Processing DeleteTablet for tablet daf72486526c4c65b04d85e514620eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:36 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36588
I20260928 15:02:36.382441 27344 tablet_service.cc:1514] Processing DeleteTablet for tablet daf72486526c4c65b04d85e514620eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:36 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260928 15:02:36.382445 27485 tablet_service.cc:1514] Processing DeleteTablet for tablet daf72486526c4c65b04d85e514620eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:36 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:53068
I20260928 15:02:36.382717 27207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:36.382879 27207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:36.383159 27737 tablet_replica.cc:330] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: stopping tablet replica
I20260928 15:02:36.383327 27737 raft_consensus.cc:2229] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:36.383541 27737 raft_consensus.cc:2258] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:36.384689 27625 tablet_service.cc:1467] Processing CreateTablet for tablet 50e018adf5594f45ae934f2ff36a0b72 (DEFAULT_TABLE table=test-table [id=3966bbbbd1b841b7975a9700b5fc5a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:36.384843 27625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e018adf5594f45ae934f2ff36a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.384863 27737 ts_tablet_manager.cc:1905] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:36.386551 27344 tablet_service.cc:1467] Processing CreateTablet for tablet 50e018adf5594f45ae934f2ff36a0b72 (DEFAULT_TABLE table=test-table [id=3966bbbbd1b841b7975a9700b5fc5a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:36.386570 27485 tablet_service.cc:1467] Processing CreateTablet for tablet 50e018adf5594f45ae934f2ff36a0b72 (DEFAULT_TABLE table=test-table [id=3966bbbbd1b841b7975a9700b5fc5a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:36.386590 27719 tablet_bootstrap.cc:492] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Bootstrap starting.
I20260928 15:02:36.386694 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e018adf5594f45ae934f2ff36a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.386698 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e018adf5594f45ae934f2ff36a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.387048 27719 tablet_bootstrap.cc:654] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.388247 27716 tablet_bootstrap.cc:492] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Bootstrap starting.
I20260928 15:02:36.388334 27714 tablet_bootstrap.cc:492] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Bootstrap starting.
I20260928 15:02:36.388737 27716 tablet_bootstrap.cc:654] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.389132 27714 tablet_bootstrap.cc:654] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.389739 27716 tablet_bootstrap.cc:492] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: No bootstrap required, opened a new log
I20260928 15:02:36.389813 27716 ts_tablet_manager.cc:1397] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:02:36.390198 27737 ts_tablet_manager.cc:1918] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:36.390187 27716 raft_consensus.cc:348] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.390426 27737 log.cc:1196] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/daf72486526c4c65b04d85e514620eec
I20260928 15:02:36.390496 27716 raft_consensus.cc:374] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.390542 27716 raft_consensus.cc:729] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec60188e274420cb5fe8dd5c12bdbaf, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.390606 27716 consensus_queue.cc:260] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.390762 27716 ts_tablet_manager.cc:1428] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:36.390852 27737 ts_tablet_manager.cc:1939] T daf72486526c4c65b04d85e514620eec P e39e368b27584db89b0774254f47fe47: Deleting consensus metadata
I20260928 15:02:36.391471 27192 catalog_manager.cc:4841] TS e39e368b27584db89b0774254f47fe47 (127.25.123.3:35877): tablet daf72486526c4c65b04d85e514620eec (table test-table [id=c7ca4e88fa104c59a8d526a0e4c0d73d]) successfully deleted
I20260928 15:02:36.391742 27714 tablet_bootstrap.cc:492] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: No bootstrap required, opened a new log
I20260928 15:02:36.392189 27714 ts_tablet_manager.cc:1397] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 15:02:36.392565 27738 tablet_replica.cc:330] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: stopping tablet replica
I20260928 15:02:36.392589 27714 raft_consensus.cc:348] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.392731 27714 raft_consensus.cc:374] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.392757 27714 raft_consensus.cc:729] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2986836c1346d19df6c41c48ef55ef, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.392804 27714 consensus_queue.cc:260] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.392958 27738 raft_consensus.cc:2229] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:36.393055 27714 ts_tablet_manager.cc:1428] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:36.393245 27738 raft_consensus.cc:2258] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:36.393391 27739 tablet_replica.cc:330] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: stopping tablet replica
I20260928 15:02:36.393532 27739 raft_consensus.cc:2229] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:36.393646 27739 raft_consensus.cc:2258] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:36.393980 27738 ts_tablet_manager.cc:1905] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:36.394111 27719 tablet_bootstrap.cc:492] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: No bootstrap required, opened a new log
I20260928 15:02:36.394176 27719 ts_tablet_manager.cc:1397] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 15:02:36.394196 27739 ts_tablet_manager.cc:1905] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:36.394352 27719 raft_consensus.cc:348] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.394443 27719 raft_consensus.cc:374] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.394466 27719 raft_consensus.cc:729] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39e368b27584db89b0774254f47fe47, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.394522 27719 consensus_queue.cc:260] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.394610 27719 ts_tablet_manager.cc:1428] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:02:36.395989 27739 ts_tablet_manager.cc:1918] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:36.396011 27738 ts_tablet_manager.cc:1918] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:36.396075 27739 log.cc:1196] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/daf72486526c4c65b04d85e514620eec
I20260928 15:02:36.396095 27738 log.cc:1196] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/daf72486526c4c65b04d85e514620eec
I20260928 15:02:36.396365 27738 ts_tablet_manager.cc:1939] T daf72486526c4c65b04d85e514620eec P 8d2986836c1346d19df6c41c48ef55ef: Deleting consensus metadata
I20260928 15:02:36.396727 27739 ts_tablet_manager.cc:1939] T daf72486526c4c65b04d85e514620eec P cec60188e274420cb5fe8dd5c12bdbaf: Deleting consensus metadata
I20260928 15:02:36.396936 27194 catalog_manager.cc:4841] TS 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2:41503): tablet daf72486526c4c65b04d85e514620eec (table test-table [id=c7ca4e88fa104c59a8d526a0e4c0d73d]) successfully deleted
I20260928 15:02:36.397190 27194 catalog_manager.cc:4841] TS cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1:44229): tablet daf72486526c4c65b04d85e514620eec (table test-table [id=c7ca4e88fa104c59a8d526a0e4c0d73d]) successfully deleted
W20260928 15:02:36.439294 27412 tablet.cc:2369] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:36.442135 27552 tablet.cc:2369] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:36.482584 27692 tablet.cc:2369] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:36.760891 27717 raft_consensus.cc:482] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:36.761004 27717 raft_consensus.cc:504] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.761430 27717 leader_election.cc:288] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1:44229), e39e368b27584db89b0774254f47fe47 (127.25.123.3:35877)
I20260928 15:02:36.765062 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.761708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:35877 (local address 127.25.123.2:45807)
0928 15:02:36.761918 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.761932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.762018 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.762188 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.762189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.762192 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.762290 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.762293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.763041 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.763043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.763979 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.763982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.764737 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.764738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.764818 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.764824 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.764900 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.764952 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":130,"threads_started":1}
I20260928 15:02:36.765182 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.761863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:45807 (local address 127.25.123.3:35877)
0928 15:02:36.761924 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.761926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.762011 (+ 85us) server_negotiation.cc:407] Connection header received
0928 15:02:36.762094 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.762095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.762107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.762161 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.762386 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.762933 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.764090 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.764652 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.764710 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.764720 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.764979 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.765094 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:02:36.765607 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.761850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:52679 (local address 127.25.123.1:44229)
0928 15:02:36.761924 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.761926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.762098 (+ 172us) server_negotiation.cc:407] Connection header received
0928 15:02:36.762167 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.762168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.762181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.762249 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.762447 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.763039 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.764555 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.765156 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.765244 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.765254 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.765477 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.765529 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:02:36.765683 27626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e018adf5594f45ae934f2ff36a0b72" candidate_uuid: "8d2986836c1346d19df6c41c48ef55ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39e368b27584db89b0774254f47fe47" is_pre_election: true
I20260928 15:02:36.765866 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.761708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44229 (local address 127.25.123.2:52679)
0928 15:02:36.761968 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.762000 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.762088 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.762265 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.762267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.762270 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.762342 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.762362 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.763356 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.763357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.764370 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.764373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.765302 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.765303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.765381 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.765385 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.765708 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.765764 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":2,"thread_start_us":166,"threads_started":1}
I20260928 15:02:36.765867 27626 raft_consensus.cc:2454] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2986836c1346d19df6c41c48ef55ef in term 0.
I20260928 15:02:36.766140 27436 leader_election.cc:302] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2986836c1346d19df6c41c48ef55ef, e39e368b27584db89b0774254f47fe47; no voters:
I20260928 15:02:36.766162 27360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e018adf5594f45ae934f2ff36a0b72" candidate_uuid: "8d2986836c1346d19df6c41c48ef55ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" is_pre_election: true
I20260928 15:02:36.766244 27360 raft_consensus.cc:2454] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2986836c1346d19df6c41c48ef55ef in term 0.
I20260928 15:02:36.766355 27717 raft_consensus.cc:2781] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:36.766410 27717 raft_consensus.cc:482] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:36.766439 27717 raft_consensus.cc:3037] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.767266 27717 raft_consensus.cc:504] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.767472 27717 leader_election.cc:288] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [CANDIDATE]: Term 1 election: Requested vote from peers cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1:44229), e39e368b27584db89b0774254f47fe47 (127.25.123.3:35877)
I20260928 15:02:36.767639 27626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e018adf5594f45ae934f2ff36a0b72" candidate_uuid: "8d2986836c1346d19df6c41c48ef55ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39e368b27584db89b0774254f47fe47"
I20260928 15:02:36.767666 27360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e018adf5594f45ae934f2ff36a0b72" candidate_uuid: "8d2986836c1346d19df6c41c48ef55ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec60188e274420cb5fe8dd5c12bdbaf"
I20260928 15:02:36.767747 27360 raft_consensus.cc:3037] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.767750 27626 raft_consensus.cc:3037] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.768363 27360 raft_consensus.cc:2454] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2986836c1346d19df6c41c48ef55ef in term 1.
I20260928 15:02:36.768618 27438 leader_election.cc:302] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2986836c1346d19df6c41c48ef55ef, cec60188e274420cb5fe8dd5c12bdbaf; no voters:
I20260928 15:02:36.768713 27717 raft_consensus.cc:2781] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:36.768949 27717 raft_consensus.cc:686] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 1 LEADER]: Becoming Leader. State: Replica: 8d2986836c1346d19df6c41c48ef55ef, State: Running, Role: LEADER
I20260928 15:02:36.769063 27626 raft_consensus.cc:2454] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2986836c1346d19df6c41c48ef55ef in term 1.
I20260928 15:02:36.769068 27717 consensus_queue.cc:237] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } }
I20260928 15:02:36.770507 27207 catalog_manager.cc:5495] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2). New cstate: current_term: 1 leader_uuid: "8d2986836c1346d19df6c41c48ef55ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:02:36 dist-test-slave-qnz0 krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:36.872663 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.861614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.25.123.62:35699)
0928 15:02:36.861780 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.861782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.861791 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:36.861818 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.861819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.861828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.861925 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:36.862577 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.863307 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.864370 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.864998 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.865096 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.868826 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:36.868827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:36.868828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:36.868836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:36.870962 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.871648 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.871649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.871650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.871729 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:36.871955 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:36.871956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:36.871957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:36.872129 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:36.872202 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.872419 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.872538 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":107,"threads_started":1}
I20260928 15:02:36.872797 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.861346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35699 (local address 127.0.0.1:54402)
0928 15:02:36.861600 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.861612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.861679 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.862039 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.862040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.862399 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:36.862461 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.862463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.863473 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.863475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.864231 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.864234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.865099 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.865099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.865155 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.866020 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:36.866025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:36.868704 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:36.871114 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:36.871136 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:36.871139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:36.871476 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:36.871809 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:36.871810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:36.871810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:36.871876 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:36.872194 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:36.872207 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:36.872260 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.872352 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.872601 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":92,"thread_start_us":97,"threads_started":1}
I20260928 15:02:36.877239 27207 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:54402:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:02:36.877599 27207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:36.880594 27344 tablet_service.cc:1467] Processing CreateTablet for tablet 028a6add9d364d608fa174c8da721ab8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0108034cd2d6468dbf5a8082ac87b943]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:36.880543 27485 tablet_service.cc:1467] Processing CreateTablet for tablet 028a6add9d364d608fa174c8da721ab8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0108034cd2d6468dbf5a8082ac87b943]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:36.880491 27625 tablet_service.cc:1467] Processing CreateTablet for tablet 028a6add9d364d608fa174c8da721ab8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0108034cd2d6468dbf5a8082ac87b943]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:36.880810 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028a6add9d364d608fa174c8da721ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.880811 27625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028a6add9d364d608fa174c8da721ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.880811 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028a6add9d364d608fa174c8da721ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:36.882530 27716 tablet_bootstrap.cc:492] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: Bootstrap starting.
I20260928 15:02:36.882766 27719 tablet_bootstrap.cc:492] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: Bootstrap starting.
I20260928 15:02:36.882928 27714 tablet_bootstrap.cc:492] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: Bootstrap starting.
I20260928 15:02:36.883148 27716 tablet_bootstrap.cc:654] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.883210 27719 tablet_bootstrap.cc:654] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.883651 27714 tablet_bootstrap.cc:654] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:36.883880 27719 tablet_bootstrap.cc:492] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: No bootstrap required, opened a new log
I20260928 15:02:36.883915 27719 ts_tablet_manager.cc:1397] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:36.884066 27719 raft_consensus.cc:348] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.884213 27719 raft_consensus.cc:374] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.884243 27719 raft_consensus.cc:729] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39e368b27584db89b0774254f47fe47, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.884449 27719 consensus_queue.cc:260] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.884522 27714 tablet_bootstrap.cc:492] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: No bootstrap required, opened a new log
I20260928 15:02:36.884588 27714 ts_tablet_manager.cc:1397] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:36.884632 27719 ts_tablet_manager.cc:1428] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:36.884671 27722 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.884755 27714 raft_consensus.cc:348] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.884836 27714 raft_consensus.cc:374] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.884881 27714 raft_consensus.cc:729] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2986836c1346d19df6c41c48ef55ef, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.884773 27722 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:36.884958 27714 consensus_queue.cc:260] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.885054 27714 ts_tablet_manager.cc:1428] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:36.885097 27717 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.885176 27717 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:36.885303 27716 tablet_bootstrap.cc:492] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: No bootstrap required, opened a new log
I20260928 15:02:36.885368 27716 ts_tablet_manager.cc:1397] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 15:02:36.885530 27716 raft_consensus.cc:348] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.885618 27716 raft_consensus.cc:374] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:36.885646 27716 raft_consensus.cc:729] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec60188e274420cb5fe8dd5c12bdbaf, State: Initialized, Role: FOLLOWER
I20260928 15:02:36.885700 27716 consensus_queue.cc:260] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.885788 27716 ts_tablet_manager.cc:1428] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 15:02:36.885864 27721 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.885929 27721 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:36.908279 27721 raft_consensus.cc:482] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:36.908459 27721 raft_consensus.cc:504] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.908866 27721 leader_election.cc:288] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2:41503), e39e368b27584db89b0774254f47fe47 (127.25.123.3:35877)
I20260928 15:02:36.916020 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.909225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:37619 (local address 127.25.123.3:35877)
0928 15:02:36.909301 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.909303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.909391 (+ 88us) server_negotiation.cc:407] Connection header received
0928 15:02:36.909444 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.909445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.909454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.910033 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.910909 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.912277 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.915015 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.915502 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.915612 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.915623 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.915868 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.915916 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:02:36.916167 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.909074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:41503 (local address 127.25.123.1:46561)
0928 15:02:36.909241 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.909253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.909330 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.910081 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.910084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.910089 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.910861 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.910865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.911797 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.911799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.914905 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.914913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.915858 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.915860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.915932 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.915936 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.916000 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.916049 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":364,"thread_start_us":79,"threads_started":1}
I20260928 15:02:36.916240 27759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.909174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:35877 (local address 127.25.123.1:37619)
0928 15:02:36.909289 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.909298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.909369 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.910301 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.910314 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.910318 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:36.910666 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.910670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.912601 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.912602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.914530 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.914534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.915616 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.915617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.915770 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.915780 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.915861 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.915917 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":528,"thread_start_us":76,"threads_started":1}
I20260928 15:02:36.916152 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.909164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:46561 (local address 127.25.123.2:41503)
0928 15:02:36.909377 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.909380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.909390 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:36.909425 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.909426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.909440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.910044 (+ 604us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:36.911124 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.911682 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.915305 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.915750 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.915859 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.915870 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.916020 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.916080 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":54,"threads_started":1}
I20260928 15:02:36.916605 27626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028a6add9d364d608fa174c8da721ab8" candidate_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39e368b27584db89b0774254f47fe47" is_pre_election: true
I20260928 15:02:36.916689 27505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028a6add9d364d608fa174c8da721ab8" candidate_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2986836c1346d19df6c41c48ef55ef" is_pre_election: true
I20260928 15:02:36.916738 27626 raft_consensus.cc:2454] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec60188e274420cb5fe8dd5c12bdbaf in term 0.
I20260928 15:02:36.916783 27505 raft_consensus.cc:2454] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec60188e274420cb5fe8dd5c12bdbaf in term 0.
I20260928 15:02:36.917152 27298 leader_election.cc:302] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2986836c1346d19df6c41c48ef55ef, cec60188e274420cb5fe8dd5c12bdbaf; no voters:
I20260928 15:02:36.917606 27721 raft_consensus.cc:2781] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:36.917683 27721 raft_consensus.cc:482] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:36.917711 27721 raft_consensus.cc:3037] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.918517 27721 raft_consensus.cc:504] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.918692 27721 leader_election.cc:288] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [CANDIDATE]: Term 1 election: Requested vote from peers 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2:41503), e39e368b27584db89b0774254f47fe47 (127.25.123.3:35877)
I20260928 15:02:36.918881 27626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028a6add9d364d608fa174c8da721ab8" candidate_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39e368b27584db89b0774254f47fe47"
I20260928 15:02:36.918910 27505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028a6add9d364d608fa174c8da721ab8" candidate_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2986836c1346d19df6c41c48ef55ef"
I20260928 15:02:36.918977 27505 raft_consensus.cc:3037] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.918999 27626 raft_consensus.cc:3037] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:36.919675 27626 raft_consensus.cc:2454] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec60188e274420cb5fe8dd5c12bdbaf in term 1.
I20260928 15:02:36.919869 27505 raft_consensus.cc:2454] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec60188e274420cb5fe8dd5c12bdbaf in term 1.
I20260928 15:02:36.920023 27298 leader_election.cc:302] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2986836c1346d19df6c41c48ef55ef, cec60188e274420cb5fe8dd5c12bdbaf; no voters:
I20260928 15:02:36.920150 27721 raft_consensus.cc:2781] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:36.920264 27721 raft_consensus.cc:686] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [term 1 LEADER]: Becoming Leader. State: Replica: cec60188e274420cb5fe8dd5c12bdbaf, State: Running, Role: LEADER
I20260928 15:02:36.920363 27721 consensus_queue.cc:237] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } }
I20260928 15:02:36.920802 27760 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: TxnStatusTablet state changed. Reason: New leader cec60188e274420cb5fe8dd5c12bdbaf. Latest consensus state: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.920926 27760 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:36.921344 27762 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:02:36.921597 27762 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:02:36.921875 27208 catalog_manager.cc:5495] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf reported cstate change: term changed from 0 to 1, leader changed from <none> to cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1). New cstate: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:36.936743 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.931011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44229 (local address 127.0.0.1:37596)
0928 15:02:36.931244 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.931262 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.931381 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.931877 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.931879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.931880 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:36.931937 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.931940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.933802 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.933803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.934882 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.934891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.935731 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.935734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.935964 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.935995 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.936491 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.936505 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.936569 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.936656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 15:02:36.936743 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.931285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.25.123.1:44229)
0928 15:02:36.931593 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.931595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.931608 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:02:36.931653 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.931654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.931679 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.931856 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:36.932024 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.933438 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.935047 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.935484 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.935727 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.936131 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.936294 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.936435 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.936609 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.936641 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260928 15:02:36.941700 27505 raft_consensus.cc:1261] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Refusing update from remote peer cec60188e274420cb5fe8dd5c12bdbaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:36.941726 27626 raft_consensus.cc:1261] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Refusing update from remote peer cec60188e274420cb5fe8dd5c12bdbaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:36.942286 27721 consensus_queue.cc:1035] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:36.942530 27721 consensus_queue.cc:1035] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:36.944067 27722 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: TxnStatusTablet state changed. Reason: New leader cec60188e274420cb5fe8dd5c12bdbaf. Latest consensus state: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.944177 27722 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:36.944535 27717 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: TxnStatusTablet state changed. Reason: New leader cec60188e274420cb5fe8dd5c12bdbaf. Latest consensus state: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.944641 27717 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:36.946069 27721 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.946177 27721 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:36.947332 27717 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.947441 27717 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P 8d2986836c1346d19df6c41c48ef55ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:36.948244 27760 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.948351 27760 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P cec60188e274420cb5fe8dd5c12bdbaf: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:36.948611 27722 tablet_replica.cc:439] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2986836c1346d19df6c41c48ef55ef" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 41503 } } peers { permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 } } peers { permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 } } }
I20260928 15:02:36.948719 27722 tablet_replica.cc:442] T 028a6add9d364d608fa174c8da721ab8 P e39e368b27584db89b0774254f47fe47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:36.953950 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.948449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.25.123.2:41503)
0928 15:02:36.948520 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.948522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.949348 (+ 826us) server_negotiation.cc:407] Connection header received
0928 15:02:36.949406 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.949407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.949434 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.949515 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:36.949699 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.950705 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.952253 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.952723 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.952835 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.953167 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.953383 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.953492 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.953793 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.953852 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:02:36.965402 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.960638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44229 (local address 127.25.123.2:50661)
0928 15:02:36.960931 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.960950 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.961019 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.961293 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.961294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.961296 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:36.961371 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.961373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.962355 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.962356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.963613 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.963621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.964431 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.964433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.964582 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.964690 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.965154 (+ 464us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.965159 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.965250 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.965315 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260928 15:02:36.965394 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.960826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:50661 (local address 127.25.123.1:44229)
0928 15:02:36.960913 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.960916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.961015 (+ 99us) server_negotiation.cc:407] Connection header received
0928 15:02:36.961070 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.961071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.961082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.961149 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:36.961460 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.962239 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.963839 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.964205 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.964312 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.964824 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.964871 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.964957 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.965258 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.965288 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:02:36.972548 27626 raft_consensus.cc:1261] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Refusing update from remote peer 8d2986836c1346d19df6c41c48ef55ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:36.972548 27360 raft_consensus.cc:1261] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Refusing update from remote peer 8d2986836c1346d19df6c41c48ef55ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:36.973639 27717 consensus_queue.cc:1035] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39e368b27584db89b0774254f47fe47" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 15:02:36.975018 27745 consensus_queue.cc:1035] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec60188e274420cb5fe8dd5c12bdbaf" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 15:02:36.991695 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.987158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:40291 (local address 127.25.123.2:41503)
0928 15:02:36.987213 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:36.987215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:36.987225 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:36.987255 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:36.987256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:36.987268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:36.987331 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:36.988463 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.989855 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.990847 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.991077 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.991169 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.991394 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.991434 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.991508 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:36.991571 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 15:02:36.991598 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 15:02:36.991714 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:36.985539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:41503 (local address 127.25.123.1:40291)
0928 15:02:36.987071 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:36.987085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:36.987159 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:36.988114 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:36.988117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:36.988121 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:36.988272 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.988274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.989997 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.989998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.990743 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:36.990746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:36.991167 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:36.991168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:36.991245 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:36.991278 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:36.991508 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:36.991510 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:02:36.991567 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 15:02:36.991619 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"thread_start_us":1429,"threads_started":1}
I20260928 15:02:37.007496 27208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:37.007652 27208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:37.008370 27208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 266be9d500b141828df9249ace196d5e: Sending DeleteTablet for 3 replicas of tablet 50e018adf5594f45ae934f2ff36a0b72
I20260928 15:02:37.008740 27344 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e018adf5594f45ae934f2ff36a0b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:37 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260928 15:02:37.008776 27625 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e018adf5594f45ae934f2ff36a0b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:37 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36588
I20260928 15:02:37.009076 27485 tablet_service.cc:1514] Processing DeleteTablet for tablet 50e018adf5594f45ae934f2ff36a0b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:37 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:53068
I20260928 15:02:37.009298 27788 tablet_replica.cc:330] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: stopping tablet replica
I20260928 15:02:37.009418 27788 raft_consensus.cc:2229] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:37.009575 27788 raft_consensus.cc:2258] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:37.010063 27788 ts_tablet_manager.cc:1905] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:37.010118 27787 tablet_replica.cc:330] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: stopping tablet replica
I20260928 15:02:37.010210 27787 raft_consensus.cc:2229] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:37.010270 27787 raft_consensus.cc:2258] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:37.011121 27787 ts_tablet_manager.cc:1905] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:37.014454 27786 tablet_replica.cc:330] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: stopping tablet replica
I20260928 15:02:37.014549 27786 raft_consensus.cc:2229] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:37.014614 27786 raft_consensus.cc:2258] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:37.014758 27787 ts_tablet_manager.cc:1918] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:37.014772 27788 ts_tablet_manager.cc:1918] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:37.015203 27787 log.cc:1196] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/50e018adf5594f45ae934f2ff36a0b72
I20260928 15:02:37.015407 27788 log.cc:1196] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/50e018adf5594f45ae934f2ff36a0b72
I20260928 15:02:37.015466 27787 ts_tablet_manager.cc:1939] T 50e018adf5594f45ae934f2ff36a0b72 P e39e368b27584db89b0774254f47fe47: Deleting consensus metadata
I20260928 15:02:37.015748 27788 ts_tablet_manager.cc:1939] T 50e018adf5594f45ae934f2ff36a0b72 P 8d2986836c1346d19df6c41c48ef55ef: Deleting consensus metadata
I20260928 15:02:37.017277 27786 ts_tablet_manager.cc:1905] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:37.019692 27194 catalog_manager.cc:4841] TS 8d2986836c1346d19df6c41c48ef55ef (127.25.123.2:41503): tablet 50e018adf5594f45ae934f2ff36a0b72 (table test-table [id=3966bbbbd1b841b7975a9700b5fc5a65]) successfully deleted
I20260928 15:02:37.021265 27192 catalog_manager.cc:4841] TS e39e368b27584db89b0774254f47fe47 (127.25.123.3:35877): tablet 50e018adf5594f45ae934f2ff36a0b72 (table test-table [id=3966bbbbd1b841b7975a9700b5fc5a65]) successfully deleted
I20260928 15:02:37.021466 27786 ts_tablet_manager.cc:1918] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:37.021539 27786 log.cc:1196] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/50e018adf5594f45ae934f2ff36a0b72
I20260928 15:02:37.021818 27786 ts_tablet_manager.cc:1939] T 50e018adf5594f45ae934f2ff36a0b72 P cec60188e274420cb5fe8dd5c12bdbaf: Deleting consensus metadata
I20260928 15:02:37.023330 27194 catalog_manager.cc:4841] TS cec60188e274420cb5fe8dd5c12bdbaf (127.25.123.1:44229): tablet 50e018adf5594f45ae934f2ff36a0b72 (table test-table [id=3966bbbbd1b841b7975a9700b5fc5a65]) successfully deleted
Sep 28 15:02:37 dist-test-slave-qnz0 krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:02:37.036247 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.010557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.25.123.1:44229)
0928 15:02:37.010607 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:37.010610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:37.014095 (+ 3485us) server_negotiation.cc:407] Connection header received
0928 15:02:37.014219 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:37.014220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:37.014246 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:37.014340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:37.020287 (+ 5947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.021122 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.022433 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.022661 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.022756 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.027291 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:37.027316 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:37.027316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:37.027326 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:37.034798 (+ 7472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.035300 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.035311 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.035312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.035376 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.035596 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.035596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.035597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.035771 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:37.035956 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:37.036005 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 15:02:37.036102 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 15:02:37.036595 27398 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37600
I20260928 15:02:37.046640 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:37.037669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.25.123.62:35699)
0928 15:02:37.037979 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:37.037981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:37.037995 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:02:37.038024 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:37.038026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:37.038040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:37.038124 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:37.038811 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.039445 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.040277 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:37.040728 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:37.040846 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:37.042243 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:37.042245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:37.042246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:37.042255 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:37.044252 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.044829 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.044831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.044831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.044910 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:37.045196 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:37.045198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:37.045198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:37.045556 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:37.045849 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:37.046349 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 15:02:37.046484 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
W20260928 15:02:37.047147 27208 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54406
I20260928 15:02:37.048478 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 27280
I20260928 15:02:37.061906 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 27420
I20260928 15:02:37.068732 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 27560
I20260928 15:02:37.079432 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 27176
2026-09-28T15:02:37Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3487 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:37 dist-test-slave-qnz0 krb5kdc[27795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:37 dist-test-slave-qnz0 krb5kdc[27795](info): set up 2 sockets
Sep 28 15:02:37 dist-test-slave-qnz0 krb5kdc[27795](info): commencing operation
krb5kdc: starting...
W20260928 15:02:38.372107 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:38 dist-test-slave-qnz0 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:38.567932 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39711
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:38.706591 27808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:38.707013 27808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:38.707042 27808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:38.708707 27808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:38.708784 27808 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:02:38.708799 27808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:38.708812 27808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:38.708825 27808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:38.708838 27808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:38.714841 27808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39711
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:38.716861 27808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:38.717306 27808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:02:38.726128 27808 server_base.cc:1034] running on GCE node
W20260928 15:02:38.726364 27813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:38.726511 27816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:38.726511 27814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:38.727074 27808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:38.727490 27808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:38.729277 27808 hybrid_clock.cc:648] HybridClock initialized: now 1790607758729251 us; error 49 us; skew 500 ppm
Sep 28 15:02:38 dist-test-slave-qnz0 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:38.734656 27808 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:38.735272 27808 webserver.cc:460] Webserver started at http://127.25.123.62:37475/ using document root <none> and password file <none>
I20260928 15:02:38.735549 27808 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:38.735610 27808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:38.735872 27808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:38.737784 27808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf"
format_stamp: "Formatted at 2026-09-28 15:02:38 on dist-test-slave-qnz0"
server_key: "be931ebc321fb0b739d290565c7b1c35"
server_key_iv: "9913eee688255990582d6930ff4eb1ec"
server_key_version: "encryptionkey@0"
I20260928 15:02:38.738678 27808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf"
format_stamp: "Formatted at 2026-09-28 15:02:38 on dist-test-slave-qnz0"
server_key: "be931ebc321fb0b739d290565c7b1c35"
server_key_iv: "9913eee688255990582d6930ff4eb1ec"
server_key_version: "encryptionkey@0"
I20260928 15:02:38.742893 27808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 15:02:38.744966 27823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:38.745669 27808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:38.745872 27808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf"
format_stamp: "Formatted at 2026-09-28 15:02:38 on dist-test-slave-qnz0"
server_key: "be931ebc321fb0b739d290565c7b1c35"
server_key_iv: "9913eee688255990582d6930ff4eb1ec"
server_key_version: "encryptionkey@0"
I20260928 15:02:38.746001 27808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:38.777324 27808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:38.782168 27808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:38.782485 27808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:38.796496 27808 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:39711
I20260928 15:02:38.796584 27885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:39711 every 8 connection(s)
I20260928 15:02:38.797293 27808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:38.798355 27886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:38.800129 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 27808
I20260928 15:02:38.800257 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:38.800489 26092 external_mini_cluster.cc:1428] Setting key 94b9349618359a9d13f8ba7c7651361f
I20260928 15:02:38.801571 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Bootstrap starting.
I20260928 15:02:38.802309 27886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:38.802642 27886 log.cc:824] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:38.803596 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: No bootstrap required, opened a new log
I20260928 15:02:38.806735 27886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39711 } }
I20260928 15:02:38.806989 27886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:38.807046 27886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d08ac0cb7a14fbbbcceeca8f3b9fecf, State: Initialized, Role: FOLLOWER
I20260928 15:02:38.807247 27886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39711 } }
I20260928 15:02:38.807355 27886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:38.807399 27886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:38.807441 27886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:38.808329 27886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39711 } }
I20260928 15:02:38.808502 27886 leader_election.cc:302] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d08ac0cb7a14fbbbcceeca8f3b9fecf; no voters:
I20260928 15:02:38.808763 27886 leader_election.cc:288] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:38.808904 27891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:38.809288 27886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:38.809340 27891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [term 1 LEADER]: Becoming Leader. State: Replica: 5d08ac0cb7a14fbbbcceeca8f3b9fecf, State: Running, Role: LEADER
Sep 28 15:02:38 dist-test-slave-qnz0 krb5kdc[27795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:38.811395 27891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39711 } }
I20260928 15:02:38.816504 27893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39711 } } }
I20260928 15:02:38.817009 27893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:38.819455 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:38.801623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.25.123.62:39711)
0928 15:02:38.801854 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:38.801865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:38.801901 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:02:38.802501 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:38.802510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:38.802576 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:38.802742 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:38.803637 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.805296 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.806405 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:38.807246 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:38.807381 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:38.812293 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:38.812300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:38.812300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:38.812322 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:38.815138 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.816067 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.816070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.816071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.816182 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:38.817329 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:38.817331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:38.817331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:38.817827 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:38.818012 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:38.818146 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 15:02:38.819233 (+ 1087us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260928 15:02:38.820024 27892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d08ac0cb7a14fbbbcceeca8f3b9fecf. Latest consensus state: current_term: 1 leader_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d08ac0cb7a14fbbbcceeca8f3b9fecf" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39711 } } }
I20260928 15:02:38.820875 27892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:38.821197 27906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:38.821280 27906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:38.821353 27907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:38.821529 27907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:38.825994 27907 catalog_manager.cc:1347] Generated new cluster ID: 5dc3d91e8f424a16aa60e910e16ed83d
I20260928 15:02:38.826099 27907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:38.853981 27907 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:38.854635 27907 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:38.857822 27907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Generated new TSK 0
I20260928 15:02:38.858084 27907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:38.927800 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39711
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:39.083871 27912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:39.084146 27912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:39.084178 27912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:39.085595 27912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:39.085664 27912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:39.085687 27912 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:02:39.085700 27912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:39.085718 27912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:39.087601 27912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:39.087879 27912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:39.088104 27912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:39.097062 27918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:39.097383 27912 server_base.cc:1034] running on GCE node
W20260928 15:02:39.097065 27920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:39.097203 27917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:39.098284 27912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:39.099747 27912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:39.101118 27912 hybrid_clock.cc:648] HybridClock initialized: now 1790607759101091 us; error 105 us; skew 500 ppm
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:39.104661 27912 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:39.105233 27912 webserver.cc:460] Webserver started at http://127.25.123.1:46849/ using document root <none> and password file <none>
I20260928 15:02:39.105504 27912 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:39.105577 27912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:39.105741 27912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:39.107303 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "b6ddc20ec3f84885bcd60391dc2d5354"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "d09cc6276960b6ccaf756e9d778cdce5"
server_key_iv: "c9df45360f35593f7a119da34e7f7eae"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.107841 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "b6ddc20ec3f84885bcd60391dc2d5354"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "d09cc6276960b6ccaf756e9d778cdce5"
server_key_iv: "c9df45360f35593f7a119da34e7f7eae"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.110256 27912 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 15:02:39.112684 27927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:39.113266 27912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 15:02:39.113466 27912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "b6ddc20ec3f84885bcd60391dc2d5354"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "d09cc6276960b6ccaf756e9d778cdce5"
server_key_iv: "c9df45360f35593f7a119da34e7f7eae"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.113605 27912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:39.145864 27912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:39.152932 27912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:39.153127 27912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:39.154488 27912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:39.154604 27912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:39.154657 27912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:39.154685 27912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:39.164753 27912 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:41997
I20260928 15:02:39.164955 28042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:41997 every 8 connection(s)
I20260928 15:02:39.165304 27912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:39.169513 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.155198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:55201 (local address 127.25.123.62:39711)
0928 15:02:39.155378 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.155382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.155510 (+ 128us) server_negotiation.cc:407] Connection header received
0928 15:02:39.156368 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.156370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.156420 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.156500 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:39.157680 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.158651 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.161000 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.161405 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.161495 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.165000 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:39.165002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:39.165002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:39.165012 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:39.167238 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.167975 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.167976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.167977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.168036 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.168384 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.168387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.168387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.168716 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:39.168824 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.169295 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.169434 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 15:02:39.169766 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.155167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39711 (local address 127.25.123.1:55201)
0928 15:02:39.155341 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.155386 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.156080 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.156620 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.156625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.157098 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:39.157504 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.157510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.158832 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.158835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.160861 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.160873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.161550 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.161551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.161748 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.162490 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:39.162504 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:39.164671 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:39.167410 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.167420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.167424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.167846 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.168115 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.168116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.168116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.168231 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.168895 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:39.168902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:39.169074 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.169210 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.169393 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260928 15:02:39.170567 28043 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39711
I20260928 15:02:39.170687 28043 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:39.170931 28043 heartbeater.cc:511] Master 127.25.123.62:39711 requested a full tablet report, sending...
I20260928 15:02:39.171540 27840 ts_manager.cc:194] Registered new tserver with Master: b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997)
I20260928 15:02:39.172286 27840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:55201
I20260928 15:02:39.173499 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 27912
I20260928 15:02:39.173589 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:39.173802 26092 external_mini_cluster.cc:1428] Setting key fab6ec0d434a9ce6855f44b75da6f6cf
I20260928 15:02:39.181360 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.171803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39711 (local address 127.25.123.1:57681)
0928 15:02:39.171932 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.171942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.171987 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.172819 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.172821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.173139 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:39.173353 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.173358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.174371 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.174373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.175406 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.175409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.175907 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.175908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.175997 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.176767 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:39.176777 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:39.177274 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:39.179656 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.179658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.179659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.179945 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.180329 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.180330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.180331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.180493 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.180977 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:39.180978 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:39.181048 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.181141 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.181270 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20260928 15:02:39.181447 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.171874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:57681 (local address 127.25.123.62:39711)
0928 15:02:39.172606 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.172610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.172629 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:02:39.172672 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.172674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.172689 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.172802 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:39.173489 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.174227 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.175517 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.175806 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.175882 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.177418 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:39.177420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:39.177420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:39.177429 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:39.179484 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.180067 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.180069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.180069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.180153 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.180584 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.180585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.180586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.180786 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:39.180892 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.181207 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.181307 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681}
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:39.220558 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39711
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:39.338642 28052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:39.338936 28052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:39.338973 28052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:39.341447 28052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:39.341537 28052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:39.341566 28052 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:02:39.341579 28052 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:39.341599 28052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:39.344226 28052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:39.344811 28052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:39.345093 28052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:39.352370 28060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:39.352384 28057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:39.352384 28058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:39.353308 28052 server_base.cc:1034] running on GCE node
I20260928 15:02:39.353600 28052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:39.353916 28052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:39.355393 28052 hybrid_clock.cc:648] HybridClock initialized: now 1790607759355341 us; error 56 us; skew 500 ppm
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:39.360507 28052 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:39.361325 28052 webserver.cc:460] Webserver started at http://127.25.123.2:40799/ using document root <none> and password file <none>
I20260928 15:02:39.361651 28052 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:39.361713 28052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:39.361832 28052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:39.363076 28052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "b480ab3f550c4d1281ceaeca30678370"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "7c85bc1e5e49909545025c61a553d933"
server_key_iv: "41476f10eac7856d0d2f18d6524abb9d"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.363480 28052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "b480ab3f550c4d1281ceaeca30678370"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "7c85bc1e5e49909545025c61a553d933"
server_key_iv: "41476f10eac7856d0d2f18d6524abb9d"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.365314 28052 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 15:02:39.366564 28067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:39.367066 28052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:02:39.367306 28052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "b480ab3f550c4d1281ceaeca30678370"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "7c85bc1e5e49909545025c61a553d933"
server_key_iv: "41476f10eac7856d0d2f18d6524abb9d"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.367412 28052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:39.388993 28052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:39.392663 28052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:39.392956 28052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:39.394331 28052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:39.394569 28052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:39.394680 28052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:39.394761 28052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:39.406999 28052 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:43281
I20260928 15:02:39.407130 28182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:43281 every 8 connection(s)
I20260928 15:02:39.407964 28052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:39.408677 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 28052
I20260928 15:02:39.408784 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:39.408999 26092 external_mini_cluster.cc:1428] Setting key 56af96347463babf6f28764b8f79f319
I20260928 15:02:39.409832 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.394990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39711 (local address 127.25.123.2:59085)
0928 15:02:39.395328 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.395363 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.395908 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.396525 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.396534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.396913 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:39.397128 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.397132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.397948 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.397949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.399186 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.399189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.399847 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.399849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.400004 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.400728 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:39.400738 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:39.403101 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:39.406806 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.406813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.406817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.407202 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.408073 (+ 871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.408075 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.408076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.408259 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.408843 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:39.408852 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:39.409064 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.409210 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.409428 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":134,"threads_started":1}
I20260928 15:02:39.410038 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.395002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:59085 (local address 127.25.123.62:39711)
0928 15:02:39.395374 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.395378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.395456 (+ 78us) server_negotiation.cc:407] Connection header received
0928 15:02:39.396064 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.396066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.396086 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.396340 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:39.397262 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.397842 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.399325 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.399707 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.399835 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.403544 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:39.403547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:39.403547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:39.403560 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:39.406631 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.407437 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.407439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.407439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.407514 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.408436 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.408438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.408439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.408699 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:39.408797 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.409307 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.409754 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 15:02:39.410547 28183 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39711
I20260928 15:02:39.410645 28183 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:39.410882 28183 heartbeater.cc:511] Master 127.25.123.62:39711 requested a full tablet report, sending...
I20260928 15:02:39.411314 27839 ts_manager.cc:194] Registered new tserver with Master: b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281)
I20260928 15:02:39.411804 27839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:59085
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:02:39.425379 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.411997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:35627 (local address 127.25.123.62:39711)
0928 15:02:39.412146 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.412149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.414185 (+ 2036us) server_negotiation.cc:407] Connection header received
0928 15:02:39.414224 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.414225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.414241 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.414335 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:39.414952 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.415560 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.417072 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.417498 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.417635 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.420433 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:39.420436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:39.420436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:39.420449 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:39.422960 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.423731 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.423733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.423733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.423807 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.424127 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.424128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.424128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.424379 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:39.424645 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.425167 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.425278 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 15:02:39.425630 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.411802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39711 (local address 127.25.123.2:35627)
0928 15:02:39.414061 (+ 2259us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.414075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.414160 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.414422 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.414424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.414711 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:39.414806 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.414808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.415803 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.415807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.416907 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.416912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.417764 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.417766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.417892 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.419381 (+ 1489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:39.419390 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:39.419999 (+ 609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:39.423167 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.423169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.423170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.423608 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.423910 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.423911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.423911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.424021 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.424763 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:39.424766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:39.424935 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.425145 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.425425 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2226,"thread_start_us":2021,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:39.460381 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39711
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:39.590698 28192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:39.590987 28192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:39.591018 28192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:39.592520 28192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:39.592577 28192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:39.592602 28192 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:02:39.592615 28192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:02:39.592634 28192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:39.594364 28192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:39.594619 28192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:39.594836 28192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:39.602339 28197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:39.602339 28198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:39.602371 28200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:39.603202 28192 server_base.cc:1034] running on GCE node
I20260928 15:02:39.603453 28192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:39.603818 28192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:39.605202 28192 hybrid_clock.cc:648] HybridClock initialized: now 1790607759605004 us; error 222 us; skew 500 ppm
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:39.609752 28192 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:39.610482 28192 webserver.cc:460] Webserver started at http://127.25.123.3:45265/ using document root <none> and password file <none>
I20260928 15:02:39.610857 28192 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:39.610958 28192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:39.611093 28192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:39.612382 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "aa1ad5e6d4d348058080f5f23523ed5f"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "342eae582a3dd1d80d80442b2f1b2fe0"
server_key_iv: "45fde2253856462556122b4c09dd2b77"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.612851 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "aa1ad5e6d4d348058080f5f23523ed5f"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "342eae582a3dd1d80d80442b2f1b2fe0"
server_key_iv: "45fde2253856462556122b4c09dd2b77"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.615212 28192 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:39.616652 28207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:39.617093 28192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:39.617185 28192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "aa1ad5e6d4d348058080f5f23523ed5f"
format_stamp: "Formatted at 2026-09-28 15:02:39 on dist-test-slave-qnz0"
server_key: "342eae582a3dd1d80d80442b2f1b2fe0"
server_key_iv: "45fde2253856462556122b4c09dd2b77"
server_key_version: "encryptionkey@0"
I20260928 15:02:39.617264 28192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:39.629784 28192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:39.632350 28192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:39.632545 28192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:39.633116 28192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:39.633162 28192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:39.633188 28192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:39.633201 28192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:39.643992 28192 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:38523
I20260928 15:02:39.644094 28322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:38523 every 8 connection(s)
I20260928 15:02:39.644539 28192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:39.648216 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.634308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:43437 (local address 127.25.123.62:39711)
0928 15:02:39.634416 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.634420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.634702 (+ 282us) server_negotiation.cc:407] Connection header received
0928 15:02:39.635753 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.635755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.635773 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.635892 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:39.637188 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.638428 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.640536 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.640903 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.640979 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.644257 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:39.644260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:39.644261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:39.644271 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:39.646265 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.646793 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.646795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.646795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.646869 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.647272 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.647274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.647274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.647417 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:39.647491 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.648060 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.648144 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:02:39.648747 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.634256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39711 (local address 127.25.123.3:43437)
0928 15:02:39.634517 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.634570 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.635583 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.636134 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.636142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.636636 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:39.636935 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.636943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.638589 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.638591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.640349 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.640354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.641013 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.641015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.641166 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.642226 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:39.642235 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:39.644136 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:39.646409 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.646416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.646418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.646668 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.646973 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.646974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.646974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.647095 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.647573 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:39.647587 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:39.647828 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.647994 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.648490 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":158,"threads_started":1}
I20260928 15:02:39.649350 28323 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39711
I20260928 15:02:39.649477 28323 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:39.649839 28323 heartbeater.cc:511] Master 127.25.123.62:39711 requested a full tablet report, sending...
I20260928 15:02:39.650348 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 28192
I20260928 15:02:39.650432 27840 ts_manager.cc:194] Registered new tserver with Master: aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523)
I20260928 15:02:39.650640 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:39.650913 26092 external_mini_cluster.cc:1428] Setting key 1e0484720017fbf227aa6e01053105ca
I20260928 15:02:39.650916 27840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:43437
I20260928 15:02:39.653614 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:39.660194 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.651479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39711 (local address 127.25.123.3:44567)
0928 15:02:39.651625 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.651635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.651696 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.651992 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.651993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.652198 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:39.652256 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.652258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.653142 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.653144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.654078 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.654084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.654740 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.654742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.654839 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.655472 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:39.655477 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:39.655984 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:39.658757 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.658758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.658759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.658974 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.659270 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:39.659271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:39.659272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:39.659334 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:39.659659 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:39.659660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:39.659718 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.659927 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.660077 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260928 15:02:39.660190 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.651471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:44567 (local address 127.25.123.62:39711)
0928 15:02:39.651514 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.651517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.651746 (+ 229us) server_negotiation.cc:407] Connection header received
0928 15:02:39.651798 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.651799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.651849 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.651946 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:39.652362 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.653041 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.654286 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.654588 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.654685 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.656244 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:39.656246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:39.656246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:39.656254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:39.658606 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.659082 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.659084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.659084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.659181 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.659432 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.659433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.659433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.659573 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:39.659669 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.659972 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.660073 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:02:39 dist-test-slave-qnz0 krb5kdc[27795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:39.692107 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.681275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.25.123.62:39711)
0928 15:02:39.681426 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.681429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.681444 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:39.681513 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.681514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.681526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.681602 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:39.682407 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.682981 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.683905 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.684201 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.684317 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.687515 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:39.687518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:39.687518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:39.687528 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:39.689242 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.689801 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.689803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.689804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.689921 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:39.690297 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:39.690297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:39.690298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:39.690463 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:39.690573 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.691681 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.691836 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 15:02:39.695086 27840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:39.697489 27840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:39.706977 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.701301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:41997 (local address 127.0.0.1:40904)
0928 15:02:39.701539 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.701586 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.701691 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.702025 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.702029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.702031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:39.702111 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.702114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.703458 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.703459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.705486 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.705490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.706370 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.706371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.706474 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.706538 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.706711 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.706872 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":730,"thread_start_us":90,"threads_started":1}
I20260928 15:02:39.707165 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.701448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.25.123.1:41997)
0928 15:02:39.701627 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.701633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.701657 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:39.701825 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.701828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.701848 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.701906 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:39.702199 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.703360 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.705600 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.706163 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.706358 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.706439 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.707005 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.707066 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260928 15:02:39.707181 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.701902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.25.123.2:43281)
0928 15:02:39.702105 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.702111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.702121 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:39.702159 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.702161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.702177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.702256 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:39.702452 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.703541 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.705619 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.706478 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.706569 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.706637 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.707023 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.707075 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260928 15:02:39.706981 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.701534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43281 (local address 127.0.0.1:44944)
0928 15:02:39.701714 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.701727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.701793 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.702323 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.702324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.702326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:39.702359 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.702361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.703596 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.703597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.705489 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.705493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.706759 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.706760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.706826 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.706830 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.706886 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.706928 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260928 15:02:39.707875 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.701452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.25.123.3:38523)
0928 15:02:39.701647 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.701659 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.701932 (+ 273us) server_negotiation.cc:407] Connection header received
0928 15:02:39.701989 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.701991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.702031 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.702313 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:39.702382 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.703174 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.705619 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.706558 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.706928 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.707000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.707028 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.707663 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260928 15:02:39.707985 28117 tablet_service.cc:1467] Processing CreateTablet for tablet f4905a7184fa4712b916d04a20bbe859 (DEFAULT_TABLE table=test-table [id=53026adea67e490b8ff6643399d190dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:39.707962 27977 tablet_service.cc:1467] Processing CreateTablet for tablet f4905a7184fa4712b916d04a20bbe859 (DEFAULT_TABLE table=test-table [id=53026adea67e490b8ff6643399d190dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:39.708457 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4905a7184fa4712b916d04a20bbe859. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:39.708467 27977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4905a7184fa4712b916d04a20bbe859. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:39.708473 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.701300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:38523 (local address 127.0.0.1:48862)
0928 15:02:39.701714 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.701727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.701793 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.702155 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.702156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.702159 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:39.702233 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.702235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.703293 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.703294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.705489 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.705493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.706659 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.706661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.706738 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.706743 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.706779 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.706832 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":91,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260928 15:02:39.710363 28257 tablet_service.cc:1467] Processing CreateTablet for tablet f4905a7184fa4712b916d04a20bbe859 (DEFAULT_TABLE table=test-table [id=53026adea67e490b8ff6643399d190dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:39.711194 28257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4905a7184fa4712b916d04a20bbe859. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:39.712657 28346 tablet_bootstrap.cc:492] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Bootstrap starting.
I20260928 15:02:39.713171 28347 tablet_bootstrap.cc:492] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Bootstrap starting.
I20260928 15:02:39.713495 28346 tablet_bootstrap.cc:654] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:39.713881 28347 tablet_bootstrap.cc:654] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:39.713953 28346 log.cc:824] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:39.714229 28347 log.cc:824] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:39.714850 28346 tablet_bootstrap.cc:492] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: No bootstrap required, opened a new log
I20260928 15:02:39.714933 28346 ts_tablet_manager.cc:1397] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:39.715039 28347 tablet_bootstrap.cc:492] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: No bootstrap required, opened a new log
I20260928 15:02:39.715104 28347 ts_tablet_manager.cc:1397] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:39.716647 28350 tablet_bootstrap.cc:492] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Bootstrap starting.
I20260928 15:02:39.717849 28350 tablet_bootstrap.cc:654] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:39.717839 28347 raft_consensus.cc:348] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.718317 28350 log.cc:824] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:39.718310 28346 raft_consensus.cc:348] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.718686 28346 raft_consensus.cc:374] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:39.718289 28347 raft_consensus.cc:374] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:39.718765 28347 raft_consensus.cc:729] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b480ab3f550c4d1281ceaeca30678370, State: Initialized, Role: FOLLOWER
I20260928 15:02:39.718755 28346 raft_consensus.cc:729] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Initialized, Role: FOLLOWER
I20260928 15:02:39.718971 28346 consensus_queue.cc:260] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.718995 28347 consensus_queue.cc:260] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.719483 28346 ts_tablet_manager.cc:1428] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 15:02:39.719511 28347 ts_tablet_manager.cc:1428] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:02:39.719652 28183 heartbeater.cc:503] Master 127.25.123.62:39711 was elected leader, sending a full tablet report...
I20260928 15:02:39.720208 28043 heartbeater.cc:503] Master 127.25.123.62:39711 was elected leader, sending a full tablet report...
I20260928 15:02:39.722033 28350 tablet_bootstrap.cc:492] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: No bootstrap required, opened a new log
I20260928 15:02:39.722137 28350 ts_tablet_manager.cc:1397] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 15:02:39.723898 28350 raft_consensus.cc:348] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.724114 28350 raft_consensus.cc:374] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:39.724159 28350 raft_consensus.cc:729] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1ad5e6d4d348058080f5f23523ed5f, State: Initialized, Role: FOLLOWER
I20260928 15:02:39.724313 28350 consensus_queue.cc:260] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.725273 28323 heartbeater.cc:503] Master 127.25.123.62:39711 was elected leader, sending a full tablet report...
I20260928 15:02:39.725322 28350 ts_tablet_manager.cc:1428] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
W20260928 15:02:39.895489 28324 tablet.cc:2369] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:39.910171 28184 tablet.cc:2369] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:39.917697 28044 tablet.cc:2369] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:39.937713 28352 raft_consensus.cc:482] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:39.937932 28352 raft_consensus.cc:504] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.938412 28352 leader_election.cc:288] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523), b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281)
I20260928 15:02:39.943035 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.939197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:36019 (local address 127.25.123.3:38523)
0928 15:02:39.939335 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.939337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.939424 (+ 87us) server_negotiation.cc:407] Connection header received
0928 15:02:39.939546 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.939547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.939557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.939624 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:39.939951 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.940732 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.942219 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.942630 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.942741 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.942752 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.942905 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.942946 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 15:02:39.943063 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.938869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:38523 (local address 127.25.123.1:36019)
0928 15:02:39.939288 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.939307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.939451 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.939649 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.939650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.939652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:39.939747 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.939750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.940939 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.940941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.942059 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.942063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.942741 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.942743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.942823 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.942827 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.942899 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.942945 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":290,"threads_started":1}
I20260928 15:02:39.943219 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.939515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:44141 (local address 127.25.123.2:43281)
0928 15:02:39.939591 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:39.939594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:39.939881 (+ 287us) server_negotiation.cc:407] Connection header received
0928 15:02:39.939998 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:39.939999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:39.940017 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:39.940322 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:39.940552 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.941048 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.942421 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.942813 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.942924 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.942934 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:39.943102 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 15:02:39.943150 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:02:39.943516 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:39.939465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43281 (local address 127.25.123.1:44141)
0928 15:02:39.939771 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:39.939782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:39.939893 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:39.940328 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:39.940329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:39.940332 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:39.940446 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.940449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.941157 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.941158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.942174 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:39.942177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:39.942927 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:39.942928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:39.943011 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:39.943014 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:02:39.943083 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 15:02:39.943127 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":238,"threads_started":1}
I20260928 15:02:39.943734 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4905a7184fa4712b916d04a20bbe859" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" is_pre_election: true
I20260928 15:02:39.943960 28277 raft_consensus.cc:2454] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 0.
I20260928 15:02:39.944008 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4905a7184fa4712b916d04a20bbe859" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370" is_pre_election: true
I20260928 15:02:39.944149 28137 raft_consensus.cc:2454] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 0.
I20260928 15:02:39.944238 27928 leader_election.cc:302] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1ad5e6d4d348058080f5f23523ed5f, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:39.944458 28352 raft_consensus.cc:2781] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:39.944541 28352 raft_consensus.cc:482] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:39.944561 28352 raft_consensus.cc:3037] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:39.945410 28352 raft_consensus.cc:504] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.945544 28352 leader_election.cc:288] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 election: Requested vote from peers aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523), b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281)
I20260928 15:02:39.946000 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4905a7184fa4712b916d04a20bbe859" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f"
I20260928 15:02:39.946167 28277 raft_consensus.cc:3037] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:39.946184 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4905a7184fa4712b916d04a20bbe859" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370"
I20260928 15:02:39.946292 28137 raft_consensus.cc:3037] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:39.947032 28277 raft_consensus.cc:2454] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 1.
I20260928 15:02:39.947276 27928 leader_election.cc:302] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1ad5e6d4d348058080f5f23523ed5f, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:39.947432 28352 raft_consensus.cc:2781] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:39.947592 28352 raft_consensus.cc:686] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 LEADER]: Becoming Leader. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Running, Role: LEADER
I20260928 15:02:39.947639 28137 raft_consensus.cc:2454] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 1.
I20260928 15:02:39.947723 28352 consensus_queue.cc:237] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } }
I20260928 15:02:39.949615 27840 catalog_manager.cc:5495] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1). New cstate: current_term: 1 leader_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } health_report { overall_health: HEALTHY } } }
Sep 28 15:02:40 dist-test-slave-qnz0 krb5kdc[27795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607758, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:40.022003 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.009380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39711 (local address 127.0.0.1:39046)
0928 15:02:40.009735 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:40.009756 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:40.010040 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:40.010444 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:40.010445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:40.010966 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:40.011044 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.011048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.012004 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.012008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.012853 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.012856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.013574 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.013576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.013665 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.014899 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:40.014904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:40.017178 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:40.020089 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:40.020107 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:40.020111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:40.020544 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:40.020911 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:40.020913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:40.020913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:40.021006 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:40.021370 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:40.021381 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:40.021451 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:02:40.021553 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:02:40.021862 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
I20260928 15:02:40.022008 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.009580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.25.123.62:39711)
0928 15:02:40.009690 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.009694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.010004 (+ 310us) server_negotiation.cc:407] Connection header received
0928 15:02:40.010301 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.010302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.010314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.010408 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:40.011228 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.011853 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.013068 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.013397 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.013504 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.017356 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:40.017359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:40.017359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:40.017369 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:40.019720 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:40.020664 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:40.020666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:40.020667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:40.020736 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:40.021120 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:40.021121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:40.021122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:40.021279 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:40.021369 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.021612 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.021929 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":233,"server-negotiator.queue_time_us":68}
I20260928 15:02:40.026050 27840 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:39046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:02:40.026373 27840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:40.029052 28117 tablet_service.cc:1467] Processing CreateTablet for tablet f32263bfe7454235a12aa73bd1d7162d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ddeb861a5db4166ae2c997d98e3a357]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:40.029122 27977 tablet_service.cc:1467] Processing CreateTablet for tablet f32263bfe7454235a12aa73bd1d7162d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ddeb861a5db4166ae2c997d98e3a357]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:40.029192 28257 tablet_service.cc:1467] Processing CreateTablet for tablet f32263bfe7454235a12aa73bd1d7162d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ddeb861a5db4166ae2c997d98e3a357]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:02:40.029372 28257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32263bfe7454235a12aa73bd1d7162d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.029372 27977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32263bfe7454235a12aa73bd1d7162d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.029376 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32263bfe7454235a12aa73bd1d7162d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.032296 28347 tablet_bootstrap.cc:492] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: Bootstrap starting.
I20260928 15:02:40.032678 28350 tablet_bootstrap.cc:492] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: Bootstrap starting.
I20260928 15:02:40.032696 28346 tablet_bootstrap.cc:492] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: Bootstrap starting.
I20260928 15:02:40.033109 28347 tablet_bootstrap.cc:654] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.033357 28346 tablet_bootstrap.cc:654] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.033535 28350 tablet_bootstrap.cc:654] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.034407 28347 tablet_bootstrap.cc:492] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: No bootstrap required, opened a new log
I20260928 15:02:40.034476 28347 ts_tablet_manager.cc:1397] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:40.034685 28347 raft_consensus.cc:348] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.034834 28347 raft_consensus.cc:374] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.034868 28347 raft_consensus.cc:729] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b480ab3f550c4d1281ceaeca30678370, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.034896 28350 tablet_bootstrap.cc:492] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: No bootstrap required, opened a new log
I20260928 15:02:40.034922 28347 consensus_queue.cc:260] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.035004 28350 ts_tablet_manager.cc:1397] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:02:40.035049 28347 ts_tablet_manager.cc:1428] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:40.035295 28350 raft_consensus.cc:348] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.035405 28350 raft_consensus.cc:374] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.035434 28350 raft_consensus.cc:729] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1ad5e6d4d348058080f5f23523ed5f, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.035501 28350 consensus_queue.cc:260] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.035624 28350 ts_tablet_manager.cc:1428] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:40.035523 28353 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.035667 28353 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:40.035820 28354 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.035921 28354 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:40.036258 28346 tablet_bootstrap.cc:492] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: No bootstrap required, opened a new log
I20260928 15:02:40.036329 28346 ts_tablet_manager.cc:1397] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 15:02:40.036667 28346 raft_consensus.cc:348] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.036775 28346 raft_consensus.cc:374] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.036799 28346 raft_consensus.cc:729] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.036844 28346 consensus_queue.cc:260] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.036972 28346 ts_tablet_manager.cc:1428] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 15:02:40.037045 28352 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.037118 28352 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:40.052380 28354 raft_consensus.cc:482] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:40.052587 28354 raft_consensus.cc:504] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.053056 28354 leader_election.cc:288] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997), b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281)
I20260928 15:02:40.057507 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.053481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:39499 (local address 127.25.123.2:43281)
0928 15:02:40.053544 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.053545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.053643 (+ 98us) server_negotiation.cc:407] Connection header received
0928 15:02:40.053696 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.053697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.053706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.053765 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:40.053932 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.054626 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.056182 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.056904 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.057003 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.057015 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.057381 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.057428 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:02:40.057507 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.053386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:39475 (local address 127.25.123.1:41997)
0928 15:02:40.053438 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.053441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.053454 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:02:40.053487 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.053488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.053500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.053575 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:40.053738 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.054528 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.056183 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.056952 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.057024 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.057033 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.057381 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.057430 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:02:40.057467 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.053260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:41997 (local address 127.25.123.3:39475)
0928 15:02:40.053329 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:40.053343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:40.053413 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:40.053582 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:40.053584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:40.053586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:40.053658 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.053660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.054690 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.054692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.056053 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.056058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.057060 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.057061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.057200 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.057202 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:02:40.057285 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:02:40.057352 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":74}
I20260928 15:02:40.057475 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.053339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43281 (local address 127.25.123.3:39499)
0928 15:02:40.053547 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:40.053558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:40.053635 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:40.053789 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:40.053790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:40.053792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:40.053854 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.053855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.054716 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.054717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.056053 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.056058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.057084 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.057085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.057197 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.057202 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:02:40.057285 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:02:40.057380 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":13,"thread_start_us":150,"threads_started":1}
I20260928 15:02:40.058229 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32263bfe7454235a12aa73bd1d7162d" candidate_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370" is_pre_election: true
I20260928 15:02:40.058365 28137 raft_consensus.cc:2454] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa1ad5e6d4d348058080f5f23523ed5f in term 0.
I20260928 15:02:40.058215 27997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32263bfe7454235a12aa73bd1d7162d" candidate_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" is_pre_election: true
I20260928 15:02:40.058538 27997 raft_consensus.cc:2454] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa1ad5e6d4d348058080f5f23523ed5f in term 0.
I20260928 15:02:40.058570 28208 leader_election.cc:302] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1ad5e6d4d348058080f5f23523ed5f, b480ab3f550c4d1281ceaeca30678370; no voters:
I20260928 15:02:40.058732 28354 raft_consensus.cc:2781] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:40.058894 28354 raft_consensus.cc:482] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:40.058924 28354 raft_consensus.cc:3037] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.060282 28354 raft_consensus.cc:504] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.060497 28354 leader_election.cc:288] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [CANDIDATE]: Term 1 election: Requested vote from peers b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997), b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281)
I20260928 15:02:40.060721 27997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32263bfe7454235a12aa73bd1d7162d" candidate_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ddc20ec3f84885bcd60391dc2d5354"
I20260928 15:02:40.060720 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32263bfe7454235a12aa73bd1d7162d" candidate_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370"
I20260928 15:02:40.060848 27997 raft_consensus.cc:3037] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.060849 28137 raft_consensus.cc:3037] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.061611 27997 raft_consensus.cc:2454] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa1ad5e6d4d348058080f5f23523ed5f in term 1.
I20260928 15:02:40.061844 28211 leader_election.cc:302] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1ad5e6d4d348058080f5f23523ed5f, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:40.061955 28137 raft_consensus.cc:2454] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa1ad5e6d4d348058080f5f23523ed5f in term 1.
I20260928 15:02:40.061985 28354 raft_consensus.cc:2781] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:40.062186 28354 raft_consensus.cc:686] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [term 1 LEADER]: Becoming Leader. State: Replica: aa1ad5e6d4d348058080f5f23523ed5f, State: Running, Role: LEADER
I20260928 15:02:40.062296 28354 consensus_queue.cc:237] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.062970 28369 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: TxnStatusTablet state changed. Reason: New leader aa1ad5e6d4d348058080f5f23523ed5f. Latest consensus state: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.063176 28369 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:40.063228 27837 catalog_manager.cc:5495] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3). New cstate: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:40.063457 28371 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:02:40.063565 28371 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:02:40.081365 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.073275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.25.123.3:38523)
0928 15:02:40.073350 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.073361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.073403 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:02:40.073552 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.073553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.073575 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.073692 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:40.073886 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.074455 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.075352 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.080001 (+ 4649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.080148 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.080467 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:40.080618 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:40.081217 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.081241 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.081272 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:02:40.081367 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.073203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:38523 (local address 127.0.0.1:48864)
0928 15:02:40.073325 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:40.073340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:40.073415 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:40.073715 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:40.073717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:40.073718 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:40.073798 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.073800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.074570 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.074571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.075247 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.075249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.080165 (+ 4916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.080168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.080257 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.080292 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:40.081122 (+ 830us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:40.081139 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 15:02:40.081204 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:02:40.081282 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 15:02:40.083551 28137 raft_consensus.cc:1261] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Refusing update from remote peer aa1ad5e6d4d348058080f5f23523ed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:40.083551 27997 raft_consensus.cc:1261] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Refusing update from remote peer aa1ad5e6d4d348058080f5f23523ed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:40.083916 28369 consensus_queue.cc:1035] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:40.084057 28354 consensus_queue.cc:1035] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:40.085671 28353 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: TxnStatusTablet state changed. Reason: New leader aa1ad5e6d4d348058080f5f23523ed5f. Latest consensus state: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.085798 28353 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:40.086030 28352 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: TxnStatusTablet state changed. Reason: New leader aa1ad5e6d4d348058080f5f23523ed5f. Latest consensus state: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.086148 28352 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:40.088009 28354 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.088126 28354 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:40.088538 28354 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.088618 28354 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P aa1ad5e6d4d348058080f5f23523ed5f: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:02:40.089037 28353 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.089145 28353 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P b480ab3f550c4d1281ceaeca30678370: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:40.096007 28376 mvcc.cc:205] Tried to move back new op lower bound from 7334329385298317312 to 7334329385215062016. Current Snapshot: MvccSnapshot[applied={T|T < 7334329385298317312}]
I20260928 15:02:40.097000 28352 tablet_replica.cc:439] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } } }
I20260928 15:02:40.097168 28352 tablet_replica.cc:442] T f32263bfe7454235a12aa73bd1d7162d P b6ddc20ec3f84885bcd60391dc2d5354: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:02:40.098460 28375 mvcc.cc:205] Tried to move back new op lower bound from 7334329385298317312 to 7334329385215062016. Current Snapshot: MvccSnapshot[applied={T|T < 7334329385298317312}]
I20260928 15:02:40.098619 28374 mvcc.cc:205] Tried to move back new op lower bound from 7334329385298317312 to 7334329385215062016. Current Snapshot: MvccSnapshot[applied={T|T < 7334329385298317312 or (T in {7334329385298317312})}]
I20260928 15:02:40.102494 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.096665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.25.123.1:41997)
0928 15:02:40.096746 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.096748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.097149 (+ 401us) server_negotiation.cc:407] Connection header received
0928 15:02:40.097185 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.097186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.097208 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.097279 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:40.097954 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.098962 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.101378 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.101608 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.101712 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.101886 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:40.102051 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:40.102148 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.102358 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.102389 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:02:40.121909 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.112656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:38311 (local address 127.25.123.3:38523)
0928 15:02:40.112741 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.112744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.113307 (+ 563us) server_negotiation.cc:407] Connection header received
0928 15:02:40.113357 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.113358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.113389 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.113482 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:40.114127 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.116508 (+ 2381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.118552 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.120525 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.120660 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.120992 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:40.121043 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:40.121105 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.121778 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.121815 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:02:40.122394 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.112130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:38523 (local address 127.25.123.1:38311)
0928 15:02:40.112703 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:40.112726 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:40.113037 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:40.113710 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:40.113711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:40.113713 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:40.114000 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.114006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.116881 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.116883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.118032 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.118041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.120711 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.120713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.120819 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.120856 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:40.121427 (+ 571us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:40.121434 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:02:40.121516 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:02:40.121774 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":472,"threads_started":1}
I20260928 15:02:40.128866 28137 raft_consensus.cc:1261] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Refusing update from remote peer b6ddc20ec3f84885bcd60391dc2d5354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:40.129088 28277 raft_consensus.cc:1261] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Refusing update from remote peer b6ddc20ec3f84885bcd60391dc2d5354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:40.130064 28357 consensus_queue.cc:1035] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:40.130422 28352 consensus_queue.cc:1035] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:40.149518 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.144766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:41997 (local address 127.25.123.3:55977)
0928 15:02:40.144843 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:40.144857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:40.144903 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:40.145237 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:40.145248 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:40.145250 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:40.145685 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.145688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.147053 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.147055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.148100 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:40.148104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.148954 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.148956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:40.149037 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.149059 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:40.149314 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:40.149317 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:02:40.149374 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 15:02:40.149431 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 15:02:40.149518 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:40.144930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:55977 (local address 127.25.123.1:41997)
0928 15:02:40.145094 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:40.145096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:40.145105 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:02:40.145130 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:40.145131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:40.145158 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:40.145209 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:40.145881 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.146952 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.148250 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:40.148837 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:40.148953 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:40.149179 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:40.149221 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:40.149314 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:40.149386 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 15:02:40.149417 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260928 15:02:40.168359 27840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:40.168538 27840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:40.169646 27840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Sending DeleteTablet for 3 replicas of tablet f4905a7184fa4712b916d04a20bbe859
I20260928 15:02:40.170490 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet f4905a7184fa4712b916d04a20bbe859 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:44944
I20260928 15:02:40.170506 27977 tablet_service.cc:1514] Processing DeleteTablet for tablet f4905a7184fa4712b916d04a20bbe859 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:40904
I20260928 15:02:40.170567 28257 tablet_service.cc:1514] Processing DeleteTablet for tablet f4905a7184fa4712b916d04a20bbe859 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:48862
I20260928 15:02:40.170822 27840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:40.171020 27840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:40.172796 28117 tablet_service.cc:1467] Processing CreateTablet for tablet 472acd9fafc14460a67ec37f4fa6de25 (DEFAULT_TABLE table=test-table [id=0a21c4584e1747e1ade87adeb147e0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:40.172919 28395 tablet_replica.cc:330] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: stopping tablet replica
I20260928 15:02:40.172960 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472acd9fafc14460a67ec37f4fa6de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.173094 28395 raft_consensus.cc:2229] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:40.173162 28395 raft_consensus.cc:2258] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:40.174054 28395 ts_tablet_manager.cc:1905] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:40.174813 28347 tablet_bootstrap.cc:492] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: Bootstrap starting.
I20260928 15:02:40.175611 28347 tablet_bootstrap.cc:654] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.176577 28393 tablet_replica.cc:330] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: stopping tablet replica
I20260928 15:02:40.176596 27977 tablet_service.cc:1467] Processing CreateTablet for tablet 472acd9fafc14460a67ec37f4fa6de25 (DEFAULT_TABLE table=test-table [id=0a21c4584e1747e1ade87adeb147e0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:40.176723 27977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472acd9fafc14460a67ec37f4fa6de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.176714 28393 raft_consensus.cc:2229] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:40.176789 28393 raft_consensus.cc:2258] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:40.176952 28394 tablet_replica.cc:330] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: stopping tablet replica
I20260928 15:02:40.177052 28394 raft_consensus.cc:2229] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:40.177160 28394 raft_consensus.cc:2258] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:40.177563 28395 ts_tablet_manager.cc:1918] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:40.177639 28395 log.cc:1196] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/f4905a7184fa4712b916d04a20bbe859
I20260928 15:02:40.178015 28395 ts_tablet_manager.cc:1939] T f4905a7184fa4712b916d04a20bbe859 P aa1ad5e6d4d348058080f5f23523ed5f: Deleting consensus metadata
I20260928 15:02:40.178328 28346 tablet_bootstrap.cc:492] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: Bootstrap starting.
I20260928 15:02:40.178625 28394 ts_tablet_manager.cc:1905] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:40.178795 28346 tablet_bootstrap.cc:654] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.179788 28393 ts_tablet_manager.cc:1905] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:40.180428 28257 tablet_service.cc:1467] Processing CreateTablet for tablet 472acd9fafc14460a67ec37f4fa6de25 (DEFAULT_TABLE table=test-table [id=0a21c4584e1747e1ade87adeb147e0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:40.180578 28257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472acd9fafc14460a67ec37f4fa6de25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:40.182130 28350 tablet_bootstrap.cc:492] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: Bootstrap starting.
I20260928 15:02:40.182173 27824 catalog_manager.cc:4841] TS aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523): tablet f4905a7184fa4712b916d04a20bbe859 (table test-table [id=53026adea67e490b8ff6643399d190dd]) successfully deleted
I20260928 15:02:40.182475 28347 tablet_bootstrap.cc:492] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: No bootstrap required, opened a new log
I20260928 15:02:40.182535 28347 ts_tablet_manager.cc:1397] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 15:02:40.182734 28347 raft_consensus.cc:348] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.182857 28347 raft_consensus.cc:374] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.182890 28347 raft_consensus.cc:729] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b480ab3f550c4d1281ceaeca30678370, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.182938 28347 consensus_queue.cc:260] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.183045 28347 ts_tablet_manager.cc:1428] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:02:40.183306 28346 tablet_bootstrap.cc:492] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: No bootstrap required, opened a new log
I20260928 15:02:40.183372 28346 ts_tablet_manager.cc:1397] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 15:02:40.183405 28350 tablet_bootstrap.cc:654] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:40.183538 28346 raft_consensus.cc:348] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.183619 28346 raft_consensus.cc:374] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.183643 28346 raft_consensus.cc:729] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.183710 28346 consensus_queue.cc:260] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.183799 28346 ts_tablet_manager.cc:1428] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:40.185936 28393 ts_tablet_manager.cc:1918] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:40.186043 28393 log.cc:1196] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/f4905a7184fa4712b916d04a20bbe859
I20260928 15:02:40.186888 28393 ts_tablet_manager.cc:1939] T f4905a7184fa4712b916d04a20bbe859 P b480ab3f550c4d1281ceaeca30678370: Deleting consensus metadata
I20260928 15:02:40.187513 27824 catalog_manager.cc:4841] TS b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281): tablet f4905a7184fa4712b916d04a20bbe859 (table test-table [id=53026adea67e490b8ff6643399d190dd]) successfully deleted
I20260928 15:02:40.187531 28394 ts_tablet_manager.cc:1918] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:40.187646 28394 log.cc:1196] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/f4905a7184fa4712b916d04a20bbe859
I20260928 15:02:40.187947 28394 ts_tablet_manager.cc:1939] T f4905a7184fa4712b916d04a20bbe859 P b6ddc20ec3f84885bcd60391dc2d5354: Deleting consensus metadata
I20260928 15:02:40.188761 27827 catalog_manager.cc:4841] TS b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997): tablet f4905a7184fa4712b916d04a20bbe859 (table test-table [id=53026adea67e490b8ff6643399d190dd]) successfully deleted
I20260928 15:02:40.194070 28350 tablet_bootstrap.cc:492] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: No bootstrap required, opened a new log
I20260928 15:02:40.194882 28350 ts_tablet_manager.cc:1397] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20260928 15:02:40.195746 28350 raft_consensus.cc:348] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.195884 28350 raft_consensus.cc:374] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:40.195907 28350 raft_consensus.cc:729] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1ad5e6d4d348058080f5f23523ed5f, State: Initialized, Role: FOLLOWER
I20260928 15:02:40.195969 28350 consensus_queue.cc:260] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.196084 28350 ts_tablet_manager.cc:1428] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:40.341248 28388 raft_consensus.cc:482] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:40.341401 28388 raft_consensus.cc:504] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.341583 28388 leader_election.cc:288] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281), aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523)
I20260928 15:02:40.342015 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472acd9fafc14460a67ec37f4fa6de25" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370" is_pre_election: true
I20260928 15:02:40.342126 28137 raft_consensus.cc:2454] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 0.
I20260928 15:02:40.342073 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472acd9fafc14460a67ec37f4fa6de25" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" is_pre_election: true
I20260928 15:02:40.342243 28277 raft_consensus.cc:2454] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 0.
I20260928 15:02:40.342311 27928 leader_election.cc:302] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b480ab3f550c4d1281ceaeca30678370, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:40.342547 28388 raft_consensus.cc:2781] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:40.342653 28388 raft_consensus.cc:482] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:40.342685 28388 raft_consensus.cc:3037] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.343978 28388 raft_consensus.cc:504] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.344158 28388 leader_election.cc:288] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 election: Requested vote from peers b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281), aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523)
I20260928 15:02:40.344416 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472acd9fafc14460a67ec37f4fa6de25" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f"
I20260928 15:02:40.344424 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472acd9fafc14460a67ec37f4fa6de25" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370"
I20260928 15:02:40.344538 28137 raft_consensus.cc:3037] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.344545 28277 raft_consensus.cc:3037] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:40.345235 28277 raft_consensus.cc:2454] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 1.
I20260928 15:02:40.345501 28137 raft_consensus.cc:2454] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 1.
I20260928 15:02:40.345543 27928 leader_election.cc:302] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1ad5e6d4d348058080f5f23523ed5f, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:40.345652 28388 raft_consensus.cc:2781] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:40.345705 28388 raft_consensus.cc:686] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 LEADER]: Becoming Leader. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Running, Role: LEADER
I20260928 15:02:40.345759 28388 consensus_queue.cc:237] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:40.346477 27840 catalog_manager.cc:5495] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1). New cstate: current_term: 1 leader_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } health_report { overall_health: UNKNOWN } } }
I20260928 15:02:40.373469 28137 raft_consensus.cc:1261] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Refusing update from remote peer b6ddc20ec3f84885bcd60391dc2d5354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:40.373450 28277 raft_consensus.cc:1261] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Refusing update from remote peer b6ddc20ec3f84885bcd60391dc2d5354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:40.373749 28357 consensus_queue.cc:1035] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:40.373867 28388 consensus_queue.cc:1035] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:02:40.381768 28376 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:02:41.921103 27837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:41.921309 27837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:41.922955 27837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Sending DeleteTablet for 3 replicas of tablet 472acd9fafc14460a67ec37f4fa6de25
I20260928 15:02:41.923296 27977 tablet_service.cc:1514] Processing DeleteTablet for tablet 472acd9fafc14460a67ec37f4fa6de25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:41 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:40904
I20260928 15:02:41.923446 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet 472acd9fafc14460a67ec37f4fa6de25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:41 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:44944
I20260928 15:02:41.923580 28423 tablet_replica.cc:330] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: stopping tablet replica
I20260928 15:02:41.923677 28257 tablet_service.cc:1514] Processing DeleteTablet for tablet 472acd9fafc14460a67ec37f4fa6de25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:41 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:48862
I20260928 15:02:41.923727 28423 raft_consensus.cc:2229] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:41.923817 28423 raft_consensus.cc:2258] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:41.923902 28425 tablet_replica.cc:330] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: stopping tablet replica
I20260928 15:02:41.923978 28425 raft_consensus.cc:2229] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:41.924033 28425 raft_consensus.cc:2258] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:41.924252 28423 ts_tablet_manager.cc:1905] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:41.924296 28425 ts_tablet_manager.cc:1905] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:41.926589 28425 ts_tablet_manager.cc:1918] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:41.926592 28423 ts_tablet_manager.cc:1918] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:41.926679 28423 log.cc:1196] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/472acd9fafc14460a67ec37f4fa6de25
I20260928 15:02:41.926677 28425 log.cc:1196] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/472acd9fafc14460a67ec37f4fa6de25
I20260928 15:02:41.926888 28424 tablet_replica.cc:330] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: stopping tablet replica
I20260928 15:02:41.926940 28425 ts_tablet_manager.cc:1939] T 472acd9fafc14460a67ec37f4fa6de25 P aa1ad5e6d4d348058080f5f23523ed5f: Deleting consensus metadata
I20260928 15:02:41.926968 28423 ts_tablet_manager.cc:1939] T 472acd9fafc14460a67ec37f4fa6de25 P b6ddc20ec3f84885bcd60391dc2d5354: Deleting consensus metadata
I20260928 15:02:41.926970 28424 raft_consensus.cc:2229] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:41.927022 28424 raft_consensus.cc:2258] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:41.927349 28424 ts_tablet_manager.cc:1905] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:41.927524 27827 catalog_manager.cc:4841] TS b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997): tablet 472acd9fafc14460a67ec37f4fa6de25 (table test-table [id=0a21c4584e1747e1ade87adeb147e0c8]) successfully deleted
I20260928 15:02:41.927546 27824 catalog_manager.cc:4841] TS aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523): tablet 472acd9fafc14460a67ec37f4fa6de25 (table test-table [id=0a21c4584e1747e1ade87adeb147e0c8]) successfully deleted
I20260928 15:02:41.929708 28424 ts_tablet_manager.cc:1918] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:41.930006 28424 log.cc:1196] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/472acd9fafc14460a67ec37f4fa6de25
I20260928 15:02:41.930287 28424 ts_tablet_manager.cc:1939] T 472acd9fafc14460a67ec37f4fa6de25 P b480ab3f550c4d1281ceaeca30678370: Deleting consensus metadata
I20260928 15:02:41.930918 27824 catalog_manager.cc:4841] TS b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281): tablet 472acd9fafc14460a67ec37f4fa6de25 (table test-table [id=0a21c4584e1747e1ade87adeb147e0c8]) successfully deleted
Sep 28 15:02:41 dist-test-slave-qnz0 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:02:41 dist-test-slave-qnz0 krb5kdc[27795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:41.964221 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:41.945393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.25.123.62:39711)
0928 15:02:41.945630 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:41.945632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:41.945644 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:02:41.945771 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:41.945772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:41.945784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:41.945934 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:41.946673 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.947244 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.948215 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:41.948558 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:41.948627 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:41.959931 (+ 11304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:41.959935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:41.959935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:41.959945 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:41.962155 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:41.962801 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:41.962802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:41.962803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:41.962883 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:41.963302 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:41.963305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:41.963305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:41.963588 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:41.963727 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:41.963986 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:02:41.964116 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20260928 15:02:41.967427 27837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:41.967895 27837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:41.970711 27977 tablet_service.cc:1467] Processing CreateTablet for tablet 789773e7d49b4939abb51bdd3e1681ac (DEFAULT_TABLE table=test-table [id=0d79751b74084c29988c19c7cbf9cb64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:41.970993 27977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789773e7d49b4939abb51bdd3e1681ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:41.970947 28117 tablet_service.cc:1467] Processing CreateTablet for tablet 789773e7d49b4939abb51bdd3e1681ac (DEFAULT_TABLE table=test-table [id=0d79751b74084c29988c19c7cbf9cb64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:41.971132 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789773e7d49b4939abb51bdd3e1681ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:41.972776 28434 tablet_bootstrap.cc:492] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: Bootstrap starting.
I20260928 15:02:41.972851 28257 tablet_service.cc:1467] Processing CreateTablet for tablet 789773e7d49b4939abb51bdd3e1681ac (DEFAULT_TABLE table=test-table [id=0d79751b74084c29988c19c7cbf9cb64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:41.972981 28257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789773e7d49b4939abb51bdd3e1681ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:41.973219 28434 tablet_bootstrap.cc:654] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:41.973489 28435 tablet_bootstrap.cc:492] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: Bootstrap starting.
I20260928 15:02:41.973888 28435 tablet_bootstrap.cc:654] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:41.974418 28438 tablet_bootstrap.cc:492] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: Bootstrap starting.
I20260928 15:02:41.974601 28435 tablet_bootstrap.cc:492] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: No bootstrap required, opened a new log
I20260928 15:02:41.974663 28435 ts_tablet_manager.cc:1397] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:41.974814 28438 tablet_bootstrap.cc:654] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:41.974810 28435 raft_consensus.cc:348] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:41.974916 28435 raft_consensus.cc:374] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:41.974951 28435 raft_consensus.cc:729] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Initialized, Role: FOLLOWER
I20260928 15:02:41.974984 28434 tablet_bootstrap.cc:492] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: No bootstrap required, opened a new log
I20260928 15:02:41.975044 28434 ts_tablet_manager.cc:1397] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:41.975032 28435 consensus_queue.cc:260] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:41.975158 28435 ts_tablet_manager.cc:1428] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:02:41.975417 28438 tablet_bootstrap.cc:492] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: No bootstrap required, opened a new log
I20260928 15:02:41.975416 28434 raft_consensus.cc:348] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:41.975505 28434 raft_consensus.cc:374] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:41.975536 28434 raft_consensus.cc:729] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b480ab3f550c4d1281ceaeca30678370, State: Initialized, Role: FOLLOWER
I20260928 15:02:41.975569 28434 consensus_queue.cc:260] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:41.975759 28434 ts_tablet_manager.cc:1428] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:41.975788 28438 ts_tablet_manager.cc:1397] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:41.976073 28438 raft_consensus.cc:348] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:41.976167 28438 raft_consensus.cc:374] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:41.976203 28438 raft_consensus.cc:729] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1ad5e6d4d348058080f5f23523ed5f, State: Initialized, Role: FOLLOWER
I20260928 15:02:41.976251 28438 consensus_queue.cc:260] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:41.976365 28438 ts_tablet_manager.cc:1428] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:42.051664 28414 raft_consensus.cc:482] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:42.051815 28414 raft_consensus.cc:504] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.052116 28414 leader_election.cc:288] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281), aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523)
I20260928 15:02:42.052425 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789773e7d49b4939abb51bdd3e1681ac" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" is_pre_election: true
I20260928 15:02:42.052536 28277 raft_consensus.cc:2454] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 0.
I20260928 15:02:42.052510 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789773e7d49b4939abb51bdd3e1681ac" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370" is_pre_election: true
I20260928 15:02:42.052662 28137 raft_consensus.cc:2454] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 0.
I20260928 15:02:42.052991 27928 leader_election.cc:302] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1ad5e6d4d348058080f5f23523ed5f, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:42.053216 28414 raft_consensus.cc:2781] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:42.053311 28414 raft_consensus.cc:482] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:42.053328 28414 raft_consensus.cc:3037] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:42.054374 28414 raft_consensus.cc:504] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.054559 28414 leader_election.cc:288] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 election: Requested vote from peers b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281), aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523)
I20260928 15:02:42.055166 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789773e7d49b4939abb51bdd3e1681ac" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f"
I20260928 15:02:42.055166 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789773e7d49b4939abb51bdd3e1681ac" candidate_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b480ab3f550c4d1281ceaeca30678370"
I20260928 15:02:42.055300 28137 raft_consensus.cc:3037] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:42.055300 28277 raft_consensus.cc:3037] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:42.056228 28137 raft_consensus.cc:2454] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 1.
I20260928 15:02:42.056255 28277 raft_consensus.cc:2454] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ddc20ec3f84885bcd60391dc2d5354 in term 1.
I20260928 15:02:42.056597 27928 leader_election.cc:302] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b480ab3f550c4d1281ceaeca30678370, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:42.056725 28414 raft_consensus.cc:2781] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:42.056979 28414 raft_consensus.cc:686] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 LEADER]: Becoming Leader. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Running, Role: LEADER
I20260928 15:02:42.057081 28414 consensus_queue.cc:237] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.057938 27837 catalog_manager.cc:5495] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1). New cstate: current_term: 1 leader_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } health_report { overall_health: UNKNOWN } } }
I20260928 15:02:42.079090 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.076279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.25.123.1:41997)
0928 15:02:42.076581 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:42.076583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:42.076589 (+ 6us) server_negotiation.cc:407] Connection header received
0928 15:02:42.076610 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:42.076610 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:42.076621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:42.076705 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:42.076888 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.077487 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.078329 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.078536 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.078616 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.078793 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:42.078831 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:42.078888 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:42.078957 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 15:02:42.078982 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":232,"threads_started":1}
I20260928 15:02:42.083631 28277 raft_consensus.cc:1261] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Refusing update from remote peer b6ddc20ec3f84885bcd60391dc2d5354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:42.083660 28137 raft_consensus.cc:1261] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Refusing update from remote peer b6ddc20ec3f84885bcd60391dc2d5354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:42.083994 28414 consensus_queue.cc:1035] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:42.084187 28418 consensus_queue.cc:1035] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:42.098737 27839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:42.098861 27839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:42.100191 27839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Sending DeleteTablet for 3 replicas of tablet 789773e7d49b4939abb51bdd3e1681ac
I20260928 15:02:42.100649 27977 tablet_service.cc:1514] Processing DeleteTablet for tablet 789773e7d49b4939abb51bdd3e1681ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:42 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:40904
I20260928 15:02:42.100658 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet 789773e7d49b4939abb51bdd3e1681ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:42 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:44944
I20260928 15:02:42.100701 28257 tablet_service.cc:1514] Processing DeleteTablet for tablet 789773e7d49b4939abb51bdd3e1681ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:42 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:48862
I20260928 15:02:42.100776 28423 tablet_replica.cc:330] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: stopping tablet replica
I20260928 15:02:42.100786 28424 tablet_replica.cc:330] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: stopping tablet replica
I20260928 15:02:42.100811 28425 tablet_replica.cc:330] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: stopping tablet replica
I20260928 15:02:42.100844 28424 raft_consensus.cc:2229] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:42.100845 28423 raft_consensus.cc:2229] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:42.100886 28424 raft_consensus.cc:2258] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:42.100886 28425 raft_consensus.cc:2229] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:42.100914 28423 raft_consensus.cc:2258] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:42.100944 28425 raft_consensus.cc:2258] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:42.101370 28425 ts_tablet_manager.cc:1905] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:42.101313 27839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:02:42.101442 28423 ts_tablet_manager.cc:1905] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 15:02:42.101500 27839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:42.103739 28423 ts_tablet_manager.cc:1918] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:42.103814 28423 log.cc:1196] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/789773e7d49b4939abb51bdd3e1681ac
I20260928 15:02:42.103858 28424 ts_tablet_manager.cc:1905] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:42.104043 28423 ts_tablet_manager.cc:1939] T 789773e7d49b4939abb51bdd3e1681ac P b6ddc20ec3f84885bcd60391dc2d5354: Deleting consensus metadata
I20260928 15:02:42.104168 27977 tablet_service.cc:1467] Processing CreateTablet for tablet a92ed108bf1549b79483b8295c652fcb (DEFAULT_TABLE table=test-table [id=5f12ef0fbe6f4c2db0f5ec9754c0f77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:42.104290 28257 tablet_service.cc:1467] Processing CreateTablet for tablet a92ed108bf1549b79483b8295c652fcb (DEFAULT_TABLE table=test-table [id=5f12ef0fbe6f4c2db0f5ec9754c0f77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:42.104341 27977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92ed108bf1549b79483b8295c652fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:42.104774 27827 catalog_manager.cc:4841] TS b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997): tablet 789773e7d49b4939abb51bdd3e1681ac (table test-table [id=0d79751b74084c29988c19c7cbf9cb64]) successfully deleted
I20260928 15:02:42.105916 28435 tablet_bootstrap.cc:492] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354: Bootstrap starting.
I20260928 15:02:42.106305 28435 tablet_bootstrap.cc:654] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:42.106621 28424 ts_tablet_manager.cc:1918] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:42.106712 28424 log.cc:1196] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/789773e7d49b4939abb51bdd3e1681ac
I20260928 15:02:42.106954 28435 tablet_bootstrap.cc:492] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354: No bootstrap required, opened a new log
I20260928 15:02:42.106976 28424 ts_tablet_manager.cc:1939] T 789773e7d49b4939abb51bdd3e1681ac P b480ab3f550c4d1281ceaeca30678370: Deleting consensus metadata
I20260928 15:02:42.107007 28435 ts_tablet_manager.cc:1397] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:42.107172 28117 tablet_service.cc:1467] Processing CreateTablet for tablet a92ed108bf1549b79483b8295c652fcb (DEFAULT_TABLE table=test-table [id=5f12ef0fbe6f4c2db0f5ec9754c0f77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:42.107147 28435 raft_consensus.cc:348] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.107239 28435 raft_consensus.cc:374] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:42.107282 28435 raft_consensus.cc:729] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ddc20ec3f84885bcd60391dc2d5354, State: Initialized, Role: FOLLOWER
I20260928 15:02:42.107328 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92ed108bf1549b79483b8295c652fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:42.107352 28435 consensus_queue.cc:260] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.107615 28435 ts_tablet_manager.cc:1428] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:42.107615 27824 catalog_manager.cc:4841] TS b480ab3f550c4d1281ceaeca30678370 (127.25.123.2:43281): tablet 789773e7d49b4939abb51bdd3e1681ac (table test-table [id=0d79751b74084c29988c19c7cbf9cb64]) successfully deleted
I20260928 15:02:42.108855 28434 tablet_bootstrap.cc:492] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: Bootstrap starting.
I20260928 15:02:42.108973 28425 ts_tablet_manager.cc:1918] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:02:42.109035 28425 log.cc:1196] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/789773e7d49b4939abb51bdd3e1681ac
I20260928 15:02:42.109282 28425 ts_tablet_manager.cc:1939] T 789773e7d49b4939abb51bdd3e1681ac P aa1ad5e6d4d348058080f5f23523ed5f: Deleting consensus metadata
I20260928 15:02:42.109477 28257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92ed108bf1549b79483b8295c652fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:42.109726 28434 tablet_bootstrap.cc:654] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:42.109845 27824 catalog_manager.cc:4841] TS aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523): tablet 789773e7d49b4939abb51bdd3e1681ac (table test-table [id=0d79751b74084c29988c19c7cbf9cb64]) successfully deleted
I20260928 15:02:42.110486 28434 tablet_bootstrap.cc:492] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: No bootstrap required, opened a new log
I20260928 15:02:42.110553 28434 ts_tablet_manager.cc:1397] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:42.110702 28434 raft_consensus.cc:348] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.110787 28434 raft_consensus.cc:374] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:42.110808 28434 raft_consensus.cc:729] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b480ab3f550c4d1281ceaeca30678370, State: Initialized, Role: FOLLOWER
I20260928 15:02:42.110857 28434 consensus_queue.cc:260] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.110971 28434 ts_tablet_manager.cc:1428] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:02:42.111151 28438 tablet_bootstrap.cc:492] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: Bootstrap starting.
I20260928 15:02:42.111976 28438 tablet_bootstrap.cc:654] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:42.112669 28438 tablet_bootstrap.cc:492] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: No bootstrap required, opened a new log
I20260928 15:02:42.112721 28438 ts_tablet_manager.cc:1397] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:42.112902 28438 raft_consensus.cc:348] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.112970 28438 raft_consensus.cc:374] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:42.112984 28438 raft_consensus.cc:729] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1ad5e6d4d348058080f5f23523ed5f, State: Initialized, Role: FOLLOWER
I20260928 15:02:42.113041 28438 consensus_queue.cc:260] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.113186 28438 ts_tablet_manager.cc:1428] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:42.186834 28440 raft_consensus.cc:482] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:42.187052 28440 raft_consensus.cc:504] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.187808 28440 leader_election.cc:288] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997), aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523)
I20260928 15:02:42.193946 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.188352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:36199 (local address 127.25.123.3:38523)
0928 15:02:42.188697 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:42.188699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:42.188711 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:02:42.188746 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:42.188747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:42.188765 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:42.189004 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:42.189292 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.190093 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.192616 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.193313 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.193410 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.193420 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:42.193793 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 15:02:42.193867 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":220,"threads_started":1}
I20260928 15:02:42.194031 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.188269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:48047 (local address 127.25.123.1:41997)
0928 15:02:42.188352 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:42.188354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:42.188366 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:02:42.188448 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:42.188449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:42.188462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:42.188817 (+ 355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:42.189294 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.190144 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.192616 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.193279 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.193409 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.193420 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:42.193795 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 15:02:42.193867 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:02:42.193929 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.188038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:41997 (local address 127.25.123.2:48047)
0928 15:02:42.188200 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:42.188214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:42.188281 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:42.188896 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:42.188897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:42.188901 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:42.189034 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.189036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.190319 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.190322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.192460 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.192467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.193464 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.193467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.193601 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.193612 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:02:42.193695 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:02:42.193810 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":162,"thread_start_us":80,"threads_started":1}
I20260928 15:02:42.194273 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.188038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:38523 (local address 127.25.123.2:36199)
0928 15:02:42.188244 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:42.188263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:42.188423 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:42.189027 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:42.189028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:42.189030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:42.189104 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.189105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.190362 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.190363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.192460 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.192467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.193466 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.193468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.193601 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.193611 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:02:42.193694 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:02:42.193764 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":110,"thread_start_us":94,"threads_started":1}
I20260928 15:02:42.194900 27997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92ed108bf1549b79483b8295c652fcb" candidate_uuid: "b480ab3f550c4d1281ceaeca30678370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" is_pre_election: true
I20260928 15:02:42.194865 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92ed108bf1549b79483b8295c652fcb" candidate_uuid: "b480ab3f550c4d1281ceaeca30678370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" is_pre_election: true
I20260928 15:02:42.195063 27997 raft_consensus.cc:2454] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b480ab3f550c4d1281ceaeca30678370 in term 0.
I20260928 15:02:42.195117 28277 raft_consensus.cc:2454] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b480ab3f550c4d1281ceaeca30678370 in term 0.
I20260928 15:02:42.195396 28071 leader_election.cc:302] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b480ab3f550c4d1281ceaeca30678370, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:42.195622 28440 raft_consensus.cc:2781] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:42.195664 28440 raft_consensus.cc:482] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:42.195688 28440 raft_consensus.cc:3037] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:42.196899 28440 raft_consensus.cc:504] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.197140 28440 leader_election.cc:288] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [CANDIDATE]: Term 1 election: Requested vote from peers b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997), aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523)
I20260928 15:02:42.197407 27997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92ed108bf1549b79483b8295c652fcb" candidate_uuid: "b480ab3f550c4d1281ceaeca30678370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ddc20ec3f84885bcd60391dc2d5354"
I20260928 15:02:42.197394 28277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92ed108bf1549b79483b8295c652fcb" candidate_uuid: "b480ab3f550c4d1281ceaeca30678370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1ad5e6d4d348058080f5f23523ed5f"
I20260928 15:02:42.197494 28277 raft_consensus.cc:3037] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:42.197518 27997 raft_consensus.cc:3037] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:42.198192 27997 raft_consensus.cc:2454] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b480ab3f550c4d1281ceaeca30678370 in term 1.
I20260928 15:02:42.198588 28277 raft_consensus.cc:2454] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b480ab3f550c4d1281ceaeca30678370 in term 1.
I20260928 15:02:42.198621 28071 leader_election.cc:302] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b480ab3f550c4d1281ceaeca30678370, b6ddc20ec3f84885bcd60391dc2d5354; no voters:
I20260928 15:02:42.198947 28440 raft_consensus.cc:2781] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:42.199168 28440 raft_consensus.cc:686] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 1 LEADER]: Becoming Leader. State: Replica: b480ab3f550c4d1281ceaeca30678370, State: Running, Role: LEADER
I20260928 15:02:42.199256 28440 consensus_queue.cc:237] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } }
I20260928 15:02:42.200229 27839 catalog_manager.cc:5495] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 reported cstate change: term changed from 0 to 1, leader changed from <none> to b480ab3f550c4d1281ceaeca30678370 (127.25.123.2). New cstate: current_term: 1 leader_uuid: "b480ab3f550c4d1281ceaeca30678370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b480ab3f550c4d1281ceaeca30678370" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 } health_report { overall_health: UNKNOWN } } }
I20260928 15:02:42.213052 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.209967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.25.123.2:43281)
0928 15:02:42.210278 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:42.210280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:42.210291 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:02:42.210359 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:42.210360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:42.210388 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:42.210521 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:42.210701 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.211307 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.212103 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.212328 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.212432 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.212615 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:42.212762 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:42.212854 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:42.212897 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 15:02:42.212922 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":129,"threads_started":1}
I20260928 15:02:42.221556 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.217085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:46183 (local address 127.25.123.3:38523)
0928 15:02:42.217211 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:42.217214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:42.217418 (+ 204us) server_negotiation.cc:407] Connection header received
0928 15:02:42.217454 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:42.217455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:42.217469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:42.217532 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:42.217777 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.218347 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.219718 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.220181 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.220575 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.220860 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:42.220918 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:42.220991 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:42.221348 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 15:02:42.221456 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:02:42.221597 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.216973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:38523 (local address 127.25.123.2:46183)
0928 15:02:42.217258 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:42.217289 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:42.217374 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:42.217592 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:42.217593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:42.217596 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:42.217667 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.217668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.218475 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.218477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.219545 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.219549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.220605 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.220608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.220699 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.220730 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:42.221008 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:42.221011 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:02:42.221317 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 15:02:42.221488 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":181,"threads_started":1}
I20260928 15:02:42.225646 28277 raft_consensus.cc:1261] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Refusing update from remote peer b480ab3f550c4d1281ceaeca30678370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:42.225927 27997 raft_consensus.cc:1261] T a92ed108bf1549b79483b8295c652fcb P b6ddc20ec3f84885bcd60391dc2d5354 [term 1 FOLLOWER]: Refusing update from remote peer b480ab3f550c4d1281ceaeca30678370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:42.226184 28453 consensus_queue.cc:1035] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1ad5e6d4d348058080f5f23523ed5f" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:42.226675 28440 consensus_queue.cc:1035] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ddc20ec3f84885bcd60391dc2d5354" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:42.244639 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.238796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43281 (local address 127.25.123.3:51117)
0928 15:02:42.239447 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:42.239460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:42.239823 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:42.240288 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:42.240289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:42.240292 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:02:42.240571 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.240574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.241725 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.241728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.242760 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:42.242766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.243882 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.243883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:42.243970 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.243978 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:02:42.244380 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:02:42.244383 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:02:42.244503 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 15:02:42.244565 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":354,"threads_started":1}
I20260928 15:02:42.244602 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:42.238907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:51117 (local address 127.25.123.2:43281)
0928 15:02:42.239356 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:42.239369 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:42.239783 (+ 414us) server_negotiation.cc:407] Connection header received
0928 15:02:42.240142 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:42.240143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:42.240156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:42.240287 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:42.240639 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.241357 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.242969 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:42.243725 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:42.243811 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:42.244146 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:42.244253 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:42.244344 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:42.244502 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 15:02:42.244532 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 15:02:42.245729 28405 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:02:43.789510 27837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:43.789769 27837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:43.791371 27837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d08ac0cb7a14fbbbcceeca8f3b9fecf: Sending DeleteTablet for 3 replicas of tablet a92ed108bf1549b79483b8295c652fcb
I20260928 15:02:43.791873 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet a92ed108bf1549b79483b8295c652fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:43 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:44944
I20260928 15:02:43.791818 28257 tablet_service.cc:1514] Processing DeleteTablet for tablet a92ed108bf1549b79483b8295c652fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:43 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:48862
I20260928 15:02:43.792063 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 27912
I20260928 15:02:43.794565 28487 tablet_replica.cc:330] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: stopping tablet replica
I20260928 15:02:43.794718 28487 raft_consensus.cc:2229] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:43.794791 28486 tablet_replica.cc:330] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: stopping tablet replica
I20260928 15:02:43.794876 28487 raft_consensus.cc:2258] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:43.794876 28486 raft_consensus.cc:2229] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:43.794942 28486 raft_consensus.cc:2258] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:43.796451 28486 ts_tablet_manager.cc:1905] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:43.796465 28487 ts_tablet_manager.cc:1905] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:43.800031 28486 ts_tablet_manager.cc:1918] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:43.800321 28486 log.cc:1196] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/a92ed108bf1549b79483b8295c652fcb
I20260928 15:02:43.800698 28486 ts_tablet_manager.cc:1939] T a92ed108bf1549b79483b8295c652fcb P aa1ad5e6d4d348058080f5f23523ed5f: Deleting consensus metadata
I20260928 15:02:43.801440 27824 catalog_manager.cc:4841] TS aa1ad5e6d4d348058080f5f23523ed5f (127.25.123.3:38523): tablet a92ed108bf1549b79483b8295c652fcb (table test-table [id=5f12ef0fbe6f4c2db0f5ec9754c0f77a]) successfully deleted
W20260928 15:02:43.803157 27827 connection.cc:535] client connection to 127.25.123.1:41997 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 15:02:43.803308 27827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:02:43.803437 28487 ts_tablet_manager.cc:1918] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:02:43.803443 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 28052
I20260928 15:02:43.803520 28487 log.cc:1196] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/a92ed108bf1549b79483b8295c652fcb
I20260928 15:02:43.803828 28487 ts_tablet_manager.cc:1939] T a92ed108bf1549b79483b8295c652fcb P b480ab3f550c4d1281ceaeca30678370: Deleting consensus metadata
I20260928 15:02:43.805809 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:43.805443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:41997 (local address 127.0.0.1:40916)
0928 15:02:43.805660 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:43.805733 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:41997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
W20260928 15:02:43.806288 27827 catalog_manager.cc:4584] TS b6ddc20ec3f84885bcd60391dc2d5354 (127.25.123.1:41997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a92ed108bf1549b79483b8295c652fcb: Network error: Client connection negotiation failed: client connection to 127.25.123.1:41997: connect: Connection refused (error 111)
I20260928 15:02:43.813540 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:43.812818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43281 (local address 127.0.0.1:44954)
0928 15:02:43.812918 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:43.813099 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.2:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 15:02:43.813499 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 28192
I20260928 15:02:43.822476 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 27808
2026-09-28T15:02:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6743 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:43 dist-test-slave-qnz0 krb5kdc[28492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:43 dist-test-slave-qnz0 krb5kdc[28492](info): set up 2 sockets
Sep 28 15:02:43 dist-test-slave-qnz0 krb5kdc[28492](info): commencing operation
krb5kdc: starting...
W20260928 15:02:45.019142 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:45 dist-test-slave-qnz0 krb5kdc[28492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:45.193233 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39719
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:45.296505 28505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:45.296761 28505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:45.296788 28505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:45.299058 28505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:45.299188 28505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:45.299208 28505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:45.299225 28505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:45.299240 28505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:45.302284 28505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:45.302665 28505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:45.302932 28505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:45.310984 28511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.311167 28513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:45.311251 28505 server_base.cc:1034] running on GCE node
W20260928 15:02:45.311050 28510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:45.311648 28505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:45.312037 28505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:45.313236 28505 hybrid_clock.cc:648] HybridClock initialized: now 1790607765313209 us; error 46 us; skew 500 ppm
Sep 28 15:02:45 dist-test-slave-qnz0 krb5kdc[28492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:45.316463 28505 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:45.317080 28505 webserver.cc:460] Webserver started at http://127.25.123.62:37965/ using document root <none> and password file <none>
I20260928 15:02:45.317382 28505 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:45.317461 28505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:45.317613 28505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:45.318810 28505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "6636d865f4ce42358e42459de70d5b53"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "4470c5a88c6039cbb49a422f8cfe55d3"
server_key_iv: "01ce557130117c3a62484184c7dd57d1"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.319218 28505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "6636d865f4ce42358e42459de70d5b53"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "4470c5a88c6039cbb49a422f8cfe55d3"
server_key_iv: "01ce557130117c3a62484184c7dd57d1"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.321082 28505 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 15:02:45.322279 28520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.322819 28505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:45.322978 28505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "6636d865f4ce42358e42459de70d5b53"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "4470c5a88c6039cbb49a422f8cfe55d3"
server_key_iv: "01ce557130117c3a62484184c7dd57d1"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.323083 28505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:45.355952 28505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:45.358536 28505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:45.358786 28505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:45.365301 28505 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:39719
I20260928 15:02:45.365360 28572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:39719 every 8 connection(s)
I20260928 15:02:45.365859 28505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:45.366990 28573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:45.369817 28573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53: Bootstrap starting.
I20260928 15:02:45.370458 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 28505
I20260928 15:02:45.370569 28573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:45.370628 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:45.370859 26092 external_mini_cluster.cc:1428] Setting key 6e5aef82a64a13e19eb06805a6d47ff9
I20260928 15:02:45.371096 28573 log.cc:824] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:45.372262 28573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53: No bootstrap required, opened a new log
I20260928 15:02:45.375502 28573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6636d865f4ce42358e42459de70d5b53" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39719 } }
I20260928 15:02:45.375691 28573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:45.375720 28573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6636d865f4ce42358e42459de70d5b53, State: Initialized, Role: FOLLOWER
I20260928 15:02:45.375989 28573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6636d865f4ce42358e42459de70d5b53" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39719 } }
I20260928 15:02:45.376119 28573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:45.376159 28573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:45.376219 28573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:45.377256 28573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6636d865f4ce42358e42459de70d5b53" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39719 } }
I20260928 15:02:45.377602 28573 leader_election.cc:302] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6636d865f4ce42358e42459de70d5b53; no voters:
I20260928 15:02:45.377858 28573 leader_election.cc:288] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:45.377964 28578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:45.378201 28573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:45.378283 28578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [term 1 LEADER]: Becoming Leader. State: Replica: 6636d865f4ce42358e42459de70d5b53, State: Running, Role: LEADER
I20260928 15:02:45.378484 28578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6636d865f4ce42358e42459de70d5b53" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39719 } }
I20260928 15:02:45.380750 28579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6636d865f4ce42358e42459de70d5b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6636d865f4ce42358e42459de70d5b53" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39719 } } }
I20260928 15:02:45.380653 28580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6636d865f4ce42358e42459de70d5b53. Latest consensus state: current_term: 1 leader_uuid: "6636d865f4ce42358e42459de70d5b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6636d865f4ce42358e42459de70d5b53" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39719 } } }
I20260928 15:02:45.380841 28579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:45.380844 28580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:45.381107 28587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 15:02:45 dist-test-slave-qnz0 krb5kdc[28492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:45.381856 28587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:45.383667 28587 catalog_manager.cc:1347] Generated new cluster ID: 94ac1400d6924ac1aa1f51891462b4b2
I20260928 15:02:45.383734 28587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:45.386482 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:45.372116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.25.123.62:39719)
0928 15:02:45.372451 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:45.372462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:45.372512 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:02:45.373033 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:45.373041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:45.373095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:45.373228 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:45.374430 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.375631 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.377148 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.377960 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.378632 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:45.382043 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:45.382049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:45.382049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:45.382066 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:45.384335 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.384920 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.384923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.384924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.385067 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.385343 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.385344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.385344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.385817 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:45.385969 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:45.386099 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 15:02:45.386256 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":156,"threads_started":1}
I20260928 15:02:45.394868 28587 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:45.396045 28587 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:45.408509 28587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6636d865f4ce42358e42459de70d5b53: Generated new TSK 0
I20260928 15:02:45.408800 28587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:45.453208 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39719
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:45.562726 28599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:45.563001 28599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:45.563035 28599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:45.565253 28599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:45.565342 28599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:45.565379 28599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:45.567987 28599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:45.568414 28599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:45.568789 28599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:45.577440 28605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:45.577858 28599 server_base.cc:1034] running on GCE node
W20260928 15:02:45.577876 28604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.578022 28607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:45.578316 28599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:45.578649 28599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:45.579859 28599 hybrid_clock.cc:648] HybridClock initialized: now 1790607765579825 us; error 37 us; skew 500 ppm
Sep 28 15:02:45 dist-test-slave-qnz0 krb5kdc[28492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:45.583815 28599 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:45.584327 28599 webserver.cc:460] Webserver started at http://127.25.123.1:39117/ using document root <none> and password file <none>
I20260928 15:02:45.584628 28599 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:45.584686 28599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:45.584792 28599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:45.585965 28599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "75ab6a808cd34c5d91dce36ebc9e3741"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "148afb6cae6618a16bd85aa81355b241"
server_key_iv: "83d3d9ecc33e4c8d263ed0b8cb038bc0"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.586309 28599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "75ab6a808cd34c5d91dce36ebc9e3741"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "148afb6cae6618a16bd85aa81355b241"
server_key_iv: "83d3d9ecc33e4c8d263ed0b8cb038bc0"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.588063 28599 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 15:02:45.589779 28614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.590039 28599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:45.590162 28599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "75ab6a808cd34c5d91dce36ebc9e3741"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "148afb6cae6618a16bd85aa81355b241"
server_key_iv: "83d3d9ecc33e4c8d263ed0b8cb038bc0"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.590358 28599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:45.606485 28599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:45.609368 28599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:45.609784 28599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:45.610066 28599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:45.610705 28599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:45.610775 28599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.610946 28599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:45.611007 28599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.622368 28599 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:33297
I20260928 15:02:45.622442 28727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:33297 every 8 connection(s)
I20260928 15:02:45.622836 28599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:45.631317 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 28599
I20260928 15:02:45.631570 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:45.631834 26092 external_mini_cluster.cc:1428] Setting key 3ea0d146844c328b41f27082397f986b
Sep 28 15:02:45 dist-test-slave-qnz0 krb5kdc[28492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:45.636776 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:45.623856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:58053 (local address 127.25.123.62:39719)
0928 15:02:45.623935 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:45.623937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:45.624195 (+ 258us) server_negotiation.cc:407] Connection header received
0928 15:02:45.624898 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:45.624899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:45.624910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:45.624983 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:45.626287 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.626927 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.629003 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.629239 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.629331 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:45.632947 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:45.632949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:45.632950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:45.632958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:45.635056 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.635576 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.635578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.635578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.635643 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.635982 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.635983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.635984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.636124 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:45.636198 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:45.636555 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 15:02:45.636662 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:02:45.636982 28730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:45.623808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39719 (local address 127.25.123.1:58053)
0928 15:02:45.624006 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:45.624056 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:45.624669 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:45.625112 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:45.625125 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:45.625799 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:45.626073 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:45.626079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.627101 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.627102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:45.628880 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:45.628883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.629342 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.629343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:45.629504 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:45.630639 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:45.630650 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:45.632828 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:45.635187 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:45.635205 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:45.635208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:45.635482 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:45.635723 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:45.635724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:45.635724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:45.635841 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:45.636203 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:45.636211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:45.636364 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 15:02:45.636509 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 15:02:45.636699 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260928 15:02:45.637718 28728 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39719
I20260928 15:02:45.637821 28728 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:45.638049 28728 heartbeater.cc:511] Master 127.25.123.62:39719 requested a full tablet report, sending...
I20260928 15:02:45.638990 28529 ts_manager.cc:194] Registered new tserver with Master: 75ab6a808cd34c5d91dce36ebc9e3741 (127.25.123.1:33297)
I20260928 15:02:45.640074 28529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:58053
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:45.679674 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39719
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:45.829059 28733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:45.829396 28733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:45.829444 28733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:45.831130 28733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:45.831243 28733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:45.831310 28733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:45.834317 28733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:45.834738 28733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:45.835023 28733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:45.841530 28739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.841530 28738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:45.841578 28741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:45.842032 28733 server_base.cc:1034] running on GCE node
I20260928 15:02:45.842909 28733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:45.843230 28733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:45.844475 28733 hybrid_clock.cc:648] HybridClock initialized: now 1790607765844417 us; error 72 us; skew 500 ppm
Sep 28 15:02:45 dist-test-slave-qnz0 krb5kdc[28492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:45.847705 28733 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:45.848351 28733 webserver.cc:460] Webserver started at http://127.25.123.2:36929/ using document root <none> and password file <none>
I20260928 15:02:45.848697 28733 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:45.848901 28733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:45.849155 28733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:45.850718 28733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "93a1aa0040ec415486c459ba49e17f7d"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "98e552e52f89d7002d56757d2be18fd3"
server_key_iv: "2d90678e5c7a54439cbdf236474c5092"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.851102 28733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "93a1aa0040ec415486c459ba49e17f7d"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "98e552e52f89d7002d56757d2be18fd3"
server_key_iv: "2d90678e5c7a54439cbdf236474c5092"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.853855 28733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 15:02:45.855340 28748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.855609 28733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:45.855980 28733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "93a1aa0040ec415486c459ba49e17f7d"
format_stamp: "Formatted at 2026-09-28 15:02:45 on dist-test-slave-qnz0"
server_key: "98e552e52f89d7002d56757d2be18fd3"
server_key_iv: "2d90678e5c7a54439cbdf236474c5092"
server_key_version: "encryptionkey@0"
I20260928 15:02:45.856354 28733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:45.897269 28733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:45.900589 28733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:45.900928 28733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:45.901227 28733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:45.901600 28733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:45.901660 28733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.901716 28733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:45.901762 28733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:45.908166 28733 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:46637
I20260928 15:02:45.908248 28861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:46637 every 8 connection(s)
I20260928 15:02:45.908720 28733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:45.909117 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 28733
I20260928 15:02:45.909227 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:45.909448 26092 external_mini_cluster.cc:1428] Setting key b2cf78cf05a3fd2a077c5f5701cba5f9
Sep 28 15:02:45 dist-test-slave-qnz0 krb5kdc[28492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607765, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:02:45.924952 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:45.910106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:59309 (local address 127.25.123.62:39719)
0928 15:02:45.910168 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:45.910171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:45.910288 (+ 117us) server_negotiation.cc:407] Connection header received
0928 15:02:45.910954 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:45.910955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:45.910966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:45.911032 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:45.912819 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.914329 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.916133 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.916535 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.916640 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:45.920667 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:45.920671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:45.920671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:45.920683 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:45.922491 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.923084 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.923086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.923086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.923152 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:45.923845 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:45.923846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:45.923847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:45.924021 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:45.924137 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:45.924702 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 15:02:45.924852 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 15:02:45.925040 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:45.909633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39719 (local address 127.25.123.2:59309)
0928 15:02:45.909899 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:45.909959 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:45.910802 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:45.911249 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:45.911255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:45.912157 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:45.912512 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:45.912522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.914708 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.914710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:45.915986 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:45.915992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:45.916733 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:45.916737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:45.916980 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:45.918134 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:45.918245 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:45.920525 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:45.922631 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:45.922644 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:45.922648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:45.922973 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:45.923316 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:45.923318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:45.923319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:45.923630 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:45.924172 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:45.924184 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:45.924437 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 15:02:45.924590 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:02:45.924765 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
I20260928 15:02:45.925989 28862 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39719
I20260928 15:02:45.926136 28862 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:45.926409 28862 heartbeater.cc:511] Master 127.25.123.62:39719 requested a full tablet report, sending...
I20260928 15:02:45.927016 28529 ts_manager.cc:194] Registered new tserver with Master: 93a1aa0040ec415486c459ba49e17f7d (127.25.123.2:46637)
I20260928 15:02:45.927456 28529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:59309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:45.954141 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39719
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:46.084029 28867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:46.084837 28867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:46.084980 28867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:46.087509 28867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:46.087639 28867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:46.087821 28867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:46.092094 28867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:46.092805 28867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:46.093094 28867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:46.100657 28873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:46.100657 28872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:46.100878 28875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:46.101599 28867 server_base.cc:1034] running on GCE node
I20260928 15:02:46.101912 28867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:46.102185 28867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:46.103457 28867 hybrid_clock.cc:648] HybridClock initialized: now 1790607766103349 us; error 120 us; skew 500 ppm
Sep 28 15:02:46 dist-test-slave-qnz0 krb5kdc[28492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607766, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:46.106594 28867 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:46.107149 28867 webserver.cc:460] Webserver started at http://127.25.123.3:42807/ using document root <none> and password file <none>
I20260928 15:02:46.107441 28867 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:46.107492 28867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:46.107625 28867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:46.108893 28867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "7fd7f99bcaac4b24b19fd8e818297c67"
format_stamp: "Formatted at 2026-09-28 15:02:46 on dist-test-slave-qnz0"
server_key: "a37eb77ccf58dc4ec37d640cc8214c14"
server_key_iv: "befe206d7b01b76597ec9b8489523183"
server_key_version: "encryptionkey@0"
I20260928 15:02:46.109297 28867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "7fd7f99bcaac4b24b19fd8e818297c67"
format_stamp: "Formatted at 2026-09-28 15:02:46 on dist-test-slave-qnz0"
server_key: "a37eb77ccf58dc4ec37d640cc8214c14"
server_key_iv: "befe206d7b01b76597ec9b8489523183"
server_key_version: "encryptionkey@0"
I20260928 15:02:46.111315 28867 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:46.113175 28882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.113884 28867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:46.114035 28867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "7fd7f99bcaac4b24b19fd8e818297c67"
format_stamp: "Formatted at 2026-09-28 15:02:46 on dist-test-slave-qnz0"
server_key: "a37eb77ccf58dc4ec37d640cc8214c14"
server_key_iv: "befe206d7b01b76597ec9b8489523183"
server_key_version: "encryptionkey@0"
I20260928 15:02:46.114151 28867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:46.171247 28867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:46.173995 28867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:46.174129 28867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:46.174496 28867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:46.174870 28867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:46.174909 28867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.174934 28867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:46.174947 28867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:46.183156 28867 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:45431
I20260928 15:02:46.183245 28995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:45431 every 8 connection(s)
I20260928 15:02:46.183741 28867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:46.188149 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 28867
I20260928 15:02:46.188550 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:46.188866 26092 external_mini_cluster.cc:1428] Setting key 89549d56e572f664e9574e26e20b663e
Sep 28 15:02:46 dist-test-slave-qnz0 krb5kdc[28492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607766, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:46.200335 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:46.184637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:47491 (local address 127.25.123.62:39719)
0928 15:02:46.184735 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:46.184738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:46.184993 (+ 255us) server_negotiation.cc:407] Connection header received
0928 15:02:46.185610 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:46.185612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:46.185631 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:46.185729 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:46.187183 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.187815 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.189808 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.190320 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.190414 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:46.194019 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:46.194022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:46.194022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:46.194033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:46.196642 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:46.197551 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:46.197553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:46.197553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:46.197645 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:46.198664 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:46.198666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:46.198667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:46.199049 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:46.199199 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:46.200126 (+ 927us) server_negotiation.cc:299] Negotiation successful
0928 15:02:46.200244 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:02:46.200774 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:46.184586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39719 (local address 127.25.123.3:47491)
0928 15:02:46.184800 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:46.184888 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:46.185444 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:46.185971 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:46.186027 (+ 56us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:46.186801 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:46.187037 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:46.187043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.187945 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.187946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:46.189652 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:46.189655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:46.190803 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:46.190804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:46.190982 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:46.191691 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:46.191701 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:46.193828 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:46.196832 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:46.196846 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:46.196849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:46.197413 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:46.197973 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:46.197974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:46.197974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:46.198226 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:46.199435 (+ 1209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:46.199448 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:46.199863 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 15:02:46.200059 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 15:02:46.200305 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":114,"threads_started":1}
I20260928 15:02:46.201946 28996 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39719
I20260928 15:02:46.202291 28996 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:46.202570 28996 heartbeater.cc:511] Master 127.25.123.62:39719 requested a full tablet report, sending...
I20260928 15:02:46.203408 28529 ts_manager.cc:194] Registered new tserver with Master: 7fd7f99bcaac4b24b19fd8e818297c67 (127.25.123.3:45431)
I20260928 15:02:46.203963 28529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:47491
I20260928 15:02:46.215472 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:02:46.230542 28577 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:46.229126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.25.123.62:39719)
0928 15:02:46.229378 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:46.229381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:46.229403 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:02:46.229515 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:46.229517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:46.229533 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:46.229761 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:46.230475 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55328: BlockingRecv error: recv got EOF from 127.0.0.1:55328 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 15:02:46.232165 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 28599
I20260928 15:02:46.237735 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 28733
I20260928 15:02:46.243324 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 28867
I20260928 15:02:46.250463 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 28505
2026-09-28T15:02:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2427 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:46 dist-test-slave-qnz0 krb5kdc[29008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:46 dist-test-slave-qnz0 krb5kdc[29008](info): set up 2 sockets
Sep 28 15:02:46 dist-test-slave-qnz0 krb5kdc[29008](info): commencing operation
krb5kdc: starting...
W20260928 15:02:47.515384 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:47 dist-test-slave-qnz0 krb5kdc[29008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:47.686532 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45727
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:47.788539 29021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:47.788834 29021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:47.788879 29021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:47.790360 29021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:47.790450 29021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:47.790477 29021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:47.790499 29021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:47.790527 29021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:47.792382 29021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:45727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:47.792805 29021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:47.793082 29021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:47.800474 29027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:47.800482 29026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:47.801182 29029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:47.801744 29021 server_base.cc:1034] running on GCE node
I20260928 15:02:47.802299 29021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:47.802802 29021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:47.804016 29021 hybrid_clock.cc:648] HybridClock initialized: now 1790607767803991 us; error 39 us; skew 500 ppm
Sep 28 15:02:47 dist-test-slave-qnz0 krb5kdc[29008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607767, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:47.814566 29021 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:47.815124 29021 webserver.cc:460] Webserver started at http://127.25.123.62:40151/ using document root <none> and password file <none>
I20260928 15:02:47.815418 29021 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:47.815472 29021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:47.815604 29021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:47.817198 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "eb3b433d7640477c966103a90edf73ec"
format_stamp: "Formatted at 2026-09-28 15:02:47 on dist-test-slave-qnz0"
server_key: "2f369ed736672c5edc2a5c6595fff204"
server_key_iv: "46aec81a93fb6c96e3ab279a1dc38078"
server_key_version: "encryptionkey@0"
I20260928 15:02:47.817596 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "eb3b433d7640477c966103a90edf73ec"
format_stamp: "Formatted at 2026-09-28 15:02:47 on dist-test-slave-qnz0"
server_key: "2f369ed736672c5edc2a5c6595fff204"
server_key_iv: "46aec81a93fb6c96e3ab279a1dc38078"
server_key_version: "encryptionkey@0"
I20260928 15:02:47.819223 29021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:47.820705 29036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:47.821086 29021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:47.821192 29021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "eb3b433d7640477c966103a90edf73ec"
format_stamp: "Formatted at 2026-09-28 15:02:47 on dist-test-slave-qnz0"
server_key: "2f369ed736672c5edc2a5c6595fff204"
server_key_iv: "46aec81a93fb6c96e3ab279a1dc38078"
server_key_version: "encryptionkey@0"
I20260928 15:02:47.821316 29021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:47.847391 29021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:47.851559 29021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:47.851742 29021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:47.857604 29021 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:45727
I20260928 15:02:47.857971 29088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:45727 every 8 connection(s)
I20260928 15:02:47.858038 29021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:47.858783 29089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:47.861836 29089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec: Bootstrap starting.
I20260928 15:02:47.862540 29089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:47.862848 29089 log.cc:824] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:47.863996 29089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec: No bootstrap required, opened a new log
I20260928 15:02:47.865296 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29021
I20260928 15:02:47.865434 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:47.865638 26092 external_mini_cluster.cc:1428] Setting key 051cb4fd1c4d0674f600764fbfd5d82e
I20260928 15:02:47.868553 29089 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3b433d7640477c966103a90edf73ec" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45727 } }
I20260928 15:02:47.868729 29089 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:47.868752 29089 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3b433d7640477c966103a90edf73ec, State: Initialized, Role: FOLLOWER
I20260928 15:02:47.868913 29089 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3b433d7640477c966103a90edf73ec" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45727 } }
I20260928 15:02:47.868990 29089 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:47.869025 29089 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:47.869074 29089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:47.869858 29089 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3b433d7640477c966103a90edf73ec" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45727 } }
I20260928 15:02:47.869997 29089 leader_election.cc:302] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb3b433d7640477c966103a90edf73ec; no voters:
I20260928 15:02:47.870231 29089 leader_election.cc:288] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:47.870358 29094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:47.870530 29089 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:47.870651 29094 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [term 1 LEADER]: Becoming Leader. State: Replica: eb3b433d7640477c966103a90edf73ec, State: Running, Role: LEADER
I20260928 15:02:47.870818 29094 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3b433d7640477c966103a90edf73ec" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45727 } }
I20260928 15:02:47.871260 29096 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb3b433d7640477c966103a90edf73ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3b433d7640477c966103a90edf73ec" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45727 } } }
I20260928 15:02:47.871368 29096 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:47.871253 29094 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb3b433d7640477c966103a90edf73ec. Latest consensus state: current_term: 1 leader_uuid: "eb3b433d7640477c966103a90edf73ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3b433d7640477c966103a90edf73ec" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45727 } } }
I20260928 15:02:47.871553 29094 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:47.871721 29103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:47.872560 29103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:47.874923 29103 catalog_manager.cc:1347] Generated new cluster ID: 1399a373fa064a49922a87531c3f47dd
I20260928 15:02:47.875000 29103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:02:47 dist-test-slave-qnz0 krb5kdc[29008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:47.881958 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:47.866316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.25.123.62:45727)
0928 15:02:47.866529 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:47.866538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:47.866568 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:02:47.867075 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:47.867083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:47.867140 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:47.867273 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:47.868287 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:47.869595 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:47.870775 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:47.871634 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:47.871733 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:47.875659 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:47.875664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:47.875664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:47.875680 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:47.878917 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:47.879737 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:47.879742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:47.879742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:47.879921 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:47.880332 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:47.880333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:47.880333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:47.881039 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:47.881344 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:47.881470 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:02:47.881673 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260928 15:02:47.884310 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:47.894399 29103 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:47.895026 29103 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:47.899121 29103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb3b433d7640477c966103a90edf73ec: Generated new TSK 0
I20260928 15:02:47.899362 29103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:47.906009 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:47.901417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.25.123.62:45727)
0928 15:02:47.901487 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:47.901489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:47.901634 (+ 145us) server_negotiation.cc:407] Connection header received
0928 15:02:47.901703 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:47.901704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:47.901718 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:47.901782 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:47.902231 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:47.903059 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:47.903896 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:47.904254 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:47.904330 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:47.905296 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:47.905298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:47.905298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:47.905571 (+ 273us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:47.905595 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:47.905603 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:47.905694 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:47.905874 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 15:02:47.905913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 15:02:47.908805 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29021
2026-09-28T15:02:47Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1656 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:48 dist-test-slave-qnz0 krb5kdc[29122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:48 dist-test-slave-qnz0 krb5kdc[29122](info): set up 2 sockets
Sep 28 15:02:48 dist-test-slave-qnz0 krb5kdc[29122](info): commencing operation
krb5kdc: starting...
W20260928 15:02:49.302270 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.371s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:49 dist-test-slave-qnz0 krb5kdc[29122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:49.489280 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35503
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:35503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:49.609041 29135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:49.609380 29135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:49.609433 29135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:49.611404 29135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:49.611500 29135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:49.611526 29135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:49.611548 29135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:49.611577 29135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:49.613639 29135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:35503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:49.613996 29135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:49.614275 29135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:49.622426 29140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:49.622403 29143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:49.622406 29141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:49.622843 29135 server_base.cc:1034] running on GCE node
I20260928 15:02:49.623229 29135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:49.623631 29135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:49.624830 29135 hybrid_clock.cc:648] HybridClock initialized: now 1790607769624770 us; error 92 us; skew 500 ppm
Sep 28 15:02:49 dist-test-slave-qnz0 krb5kdc[29122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607769, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:49.627856 29135 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:49.628537 29135 webserver.cc:460] Webserver started at http://127.25.123.62:34705/ using document root <none> and password file <none>
I20260928 15:02:49.628959 29135 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:49.629029 29135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:49.629172 29135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:49.630450 29135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "ec474858c4674554b08faace12733a1c"
format_stamp: "Formatted at 2026-09-28 15:02:49 on dist-test-slave-qnz0"
server_key: "87cadb2de8b94dd77d7079e3d8fb66f3"
server_key_iv: "ed682dcc9de5865a23ae8e8edd3e73c6"
server_key_version: "encryptionkey@0"
I20260928 15:02:49.631217 29135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "ec474858c4674554b08faace12733a1c"
format_stamp: "Formatted at 2026-09-28 15:02:49 on dist-test-slave-qnz0"
server_key: "87cadb2de8b94dd77d7079e3d8fb66f3"
server_key_iv: "ed682dcc9de5865a23ae8e8edd3e73c6"
server_key_version: "encryptionkey@0"
I20260928 15:02:49.633613 29135 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260928 15:02:49.634824 29150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:49.635226 29135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:49.635408 29135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "ec474858c4674554b08faace12733a1c"
format_stamp: "Formatted at 2026-09-28 15:02:49 on dist-test-slave-qnz0"
server_key: "87cadb2de8b94dd77d7079e3d8fb66f3"
server_key_iv: "ed682dcc9de5865a23ae8e8edd3e73c6"
server_key_version: "encryptionkey@0"
I20260928 15:02:49.635598 29135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:49.649875 29135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:49.653992 29135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:49.654250 29135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:49.662712 29135 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:35503
I20260928 15:02:49.662822 29202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:35503 every 8 connection(s)
I20260928 15:02:49.663213 29135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:49.664897 29203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:49.668623 29203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c: Bootstrap starting.
I20260928 15:02:49.668808 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29135
I20260928 15:02:49.668917 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:49.669121 26092 external_mini_cluster.cc:1428] Setting key ade0f107c29367fd575a53c9f2d14cd9
I20260928 15:02:49.669523 29203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:49.670104 29203 log.cc:824] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:49.671053 29203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c: No bootstrap required, opened a new log
I20260928 15:02:49.674446 29203 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec474858c4674554b08faace12733a1c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35503 } }
I20260928 15:02:49.674823 29203 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:49.674887 29203 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec474858c4674554b08faace12733a1c, State: Initialized, Role: FOLLOWER
I20260928 15:02:49.675606 29203 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec474858c4674554b08faace12733a1c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35503 } }
I20260928 15:02:49.675894 29203 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:49.675951 29203 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:49.675999 29203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:49.677423 29203 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec474858c4674554b08faace12733a1c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35503 } }
I20260928 15:02:49.677701 29203 leader_election.cc:302] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec474858c4674554b08faace12733a1c; no voters:
I20260928 15:02:49.678140 29203 leader_election.cc:288] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:49.678665 29208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:49.679065 29203 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:49.679250 29208 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [term 1 LEADER]: Becoming Leader. State: Replica: ec474858c4674554b08faace12733a1c, State: Running, Role: LEADER
I20260928 15:02:49.679493 29208 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec474858c4674554b08faace12733a1c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35503 } }
I20260928 15:02:49.680909 29209 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec474858c4674554b08faace12733a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec474858c4674554b08faace12733a1c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35503 } } }
I20260928 15:02:49.681071 29209 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [sys.catalog]: This master's current role is: LEADER
Sep 28 15:02:49 dist-test-slave-qnz0 krb5kdc[29122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:49.682551 29210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec474858c4674554b08faace12733a1c. Latest consensus state: current_term: 1 leader_uuid: "ec474858c4674554b08faace12733a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec474858c4674554b08faace12733a1c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35503 } } }
I20260928 15:02:49.682597 29219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:49.683059 29210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:49.684790 29219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:49.686950 29219 catalog_manager.cc:1347] Generated new cluster ID: 3065238da9d24c23b2d516801d9d339f
I20260928 15:02:49.687016 29219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:49.691136 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:49.669802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.25.123.62:35503)
0928 15:02:49.670084 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:49.670115 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:49.670163 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:02:49.670882 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:49.670892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:49.670959 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:49.671145 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:49.671935 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:49.674045 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:49.675596 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:49.676657 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:49.676810 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:49.683119 (+ 6309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:49.683272 (+ 153us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:49.683273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:49.683292 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:49.685913 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:49.686706 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:49.686710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:49.686711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:49.686851 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:49.689292 (+ 2441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:49.689296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:49.689297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:49.689863 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:49.690148 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:49.690333 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 15:02:49.690831 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260928 15:02:49.700481 29219 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:49.701371 29219 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:49.716208 29219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec474858c4674554b08faace12733a1c: Generated new TSK 0
I20260928 15:02:49.716530 29219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:49.761165 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35503
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:49.886092 29230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:49.886371 29230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:49.886405 29230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:49.887885 29230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:49.887969 29230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:49.888008 29230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:49.890547 29230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:49.892529 29230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:49.892899 29230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:49.903271 29235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:49.903366 29238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:49.903623 29230 server_base.cc:1034] running on GCE node
W20260928 15:02:49.903366 29236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:49.904023 29230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:49.904462 29230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:49.905660 29230 hybrid_clock.cc:648] HybridClock initialized: now 1790607769905620 us; error 57 us; skew 500 ppm
Sep 28 15:02:49 dist-test-slave-qnz0 krb5kdc[29122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607769, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:49.909637 29230 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:49.910282 29230 webserver.cc:460] Webserver started at http://127.25.123.1:46261/ using document root <none> and password file <none>
I20260928 15:02:49.910702 29230 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:49.910897 29230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:49.911180 29230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:49.912909 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "202a8133f439473ca9d7ae1ef179af77"
format_stamp: "Formatted at 2026-09-28 15:02:49 on dist-test-slave-qnz0"
server_key: "44ef0dcc1f30f24e8d9a8800fb5faff6"
server_key_iv: "3db55812d4c37aeec97eab03cd48477c"
server_key_version: "encryptionkey@0"
I20260928 15:02:49.913578 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "202a8133f439473ca9d7ae1ef179af77"
format_stamp: "Formatted at 2026-09-28 15:02:49 on dist-test-slave-qnz0"
server_key: "44ef0dcc1f30f24e8d9a8800fb5faff6"
server_key_iv: "3db55812d4c37aeec97eab03cd48477c"
server_key_version: "encryptionkey@0"
I20260928 15:02:49.917088 29230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:02:49.919461 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:49.920760 29230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 15:02:49.920926 29230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "202a8133f439473ca9d7ae1ef179af77"
format_stamp: "Formatted at 2026-09-28 15:02:49 on dist-test-slave-qnz0"
server_key: "44ef0dcc1f30f24e8d9a8800fb5faff6"
server_key_iv: "3db55812d4c37aeec97eab03cd48477c"
server_key_version: "encryptionkey@0"
I20260928 15:02:49.921021 29230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:49.950977 29230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:49.954116 29230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:49.954272 29230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:49.954890 29230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:49.957481 29230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:49.957569 29230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:49.957602 29230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:49.957614 29230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:49.969419 29230 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:34183
I20260928 15:02:49.969635 29360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:34183 every 8 connection(s)
I20260928 15:02:49.969928 29230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:49.970573 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29230
I20260928 15:02:49.970692 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:49.970940 26092 external_mini_cluster.cc:1428] Setting key 6ec527e6351ad864a7b0a22ad17585dc
Sep 28 15:02:49 dist-test-slave-qnz0 krb5kdc[29122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607769, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:02:49.990240 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:49.971514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:50403 (local address 127.25.123.62:35503)
0928 15:02:49.971721 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:49.971723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:49.971738 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:49.974318 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:49.974320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:49.974335 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:49.974406 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:49.976589 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:49.978611 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:49.981008 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:49.981317 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:49.981397 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:49.985542 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:49.985550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:49.985551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:49.985561 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:49.987474 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:49.988384 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:49.988432 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:49.988433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:49.988550 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:49.989187 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:49.989188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:49.989188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:49.989350 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:49.989452 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:49.989819 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 15:02:49.989935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 15:02:49.990408 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:49.970910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35503 (local address 127.25.123.1:50403)
0928 15:02:49.971594 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:49.971634 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:49.974060 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:49.974536 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:49.974543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:49.975494 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:49.975963 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:49.975972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:49.979033 (+ 3061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:49.979035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:49.980778 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:49.980786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:49.981444 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:49.981445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:49.981647 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:49.982900 (+ 1253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:49.982916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:49.985398 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:49.987634 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:49.987648 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:49.987651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:49.988203 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:49.988942 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:49.988943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:49.988944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:49.989100 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:49.989450 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:49.989459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:49.989652 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 15:02:49.989773 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 15:02:49.989971 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":266,"threads_started":1}
I20260928 15:02:49.993494 29361 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
I20260928 15:02:49.993788 29361 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:49.994527 29361 heartbeater.cc:511] Master 127.25.123.62:35503 requested a full tablet report, sending...
I20260928 15:02:49.996129 29167 ts_manager.cc:194] Registered new tserver with Master: 202a8133f439473ca9d7ae1ef179af77 (127.25.123.1:34183)
I20260928 15:02:50.001114 29167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:50403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:50.034637 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35503
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:50.148535 29366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:50.148996 29366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:50.149217 29366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:50.152110 29366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:50.152216 29366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:50.152261 29366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:50.154597 29366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:50.154915 29366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:50.155150 29366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:50.164160 29374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.164140 29372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.164132 29371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:50.165038 29366 server_base.cc:1034] running on GCE node
I20260928 15:02:50.165725 29366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:50.166178 29366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:50.167477 29366 hybrid_clock.cc:648] HybridClock initialized: now 1790607770167423 us; error 80 us; skew 500 ppm
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:50.172099 29366 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:50.172777 29366 webserver.cc:460] Webserver started at http://127.25.123.2:36967/ using document root <none> and password file <none>
I20260928 15:02:50.173063 29366 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:50.173141 29366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:50.173314 29366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:50.174588 29366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "a4bb343c03e544a0a50e7d3b3d25d45a"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-qnz0"
server_key: "28b5b5bcc901b2e54a817734e1802fbd"
server_key_iv: "46a26f4baafbb823af4c7d37aa3cc762"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.175127 29366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "a4bb343c03e544a0a50e7d3b3d25d45a"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-qnz0"
server_key: "28b5b5bcc901b2e54a817734e1802fbd"
server_key_iv: "46a26f4baafbb823af4c7d37aa3cc762"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.177136 29366 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 15:02:50.178732 29381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.179287 29366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:50.179440 29366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "a4bb343c03e544a0a50e7d3b3d25d45a"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-qnz0"
server_key: "28b5b5bcc901b2e54a817734e1802fbd"
server_key_iv: "46a26f4baafbb823af4c7d37aa3cc762"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.179543 29366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:50.192656 29366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:50.196049 29366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:50.196251 29366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:50.196641 29366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:50.197063 29366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:50.197124 29366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.197176 29366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:50.197212 29366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.206445 29366 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:45839
I20260928 15:02:50.206499 29495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:45839 every 8 connection(s)
I20260928 15:02:50.207077 29366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:50.213186 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29366
I20260928 15:02:50.213325 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:50.213572 26092 external_mini_cluster.cc:1428] Setting key 029f9f96e32b98cf60ab5d1ecbaa0597
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:50.222929 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.208351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:44027 (local address 127.25.123.62:35503)
0928 15:02:50.208494 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:50.208497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:50.209198 (+ 701us) server_negotiation.cc:407] Connection header received
0928 15:02:50.209893 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:50.209895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:50.209909 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:50.209979 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:50.211399 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.211942 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.213516 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.213856 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.213903 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.218409 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:50.218411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:50.218411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:50.218421 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:50.220726 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.221652 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.221655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.221655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.221770 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.222119 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.222120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.222121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.222283 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:50.222362 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:50.222708 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 15:02:50.222825 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 15:02:50.223176 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.208348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35503 (local address 127.25.123.2:44027)
0928 15:02:50.209010 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:50.209056 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:50.209647 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:50.210113 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:50.210119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:50.210944 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:50.211188 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:50.211194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.212080 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.212082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:50.213358 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:50.213363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.214002 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.214003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:50.214158 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.215354 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:50.215365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:50.217946 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:50.220869 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:50.220889 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:50.220894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:50.221460 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:50.221863 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:50.221864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:50.221865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:50.222032 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:50.222362 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:50.222371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:50.222537 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 15:02:50.222668 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:02:50.222866 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
I20260928 15:02:50.224076 29496 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35503
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:02:50.224324 29496 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:50.224627 29496 heartbeater.cc:511] Master 127.25.123.62:35503 requested a full tablet report, sending...
I20260928 15:02:50.225152 29167 ts_manager.cc:194] Registered new tserver with Master: a4bb343c03e544a0a50e7d3b3d25d45a (127.25.123.2:45839)
I20260928 15:02:50.225541 29167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:44027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:50.249269 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35503
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:50.362718 29501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:50.362985 29501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:50.363016 29501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:50.364348 29501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:50.364463 29501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:50.364511 29501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:50.367458 29501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:50.367810 29501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:50.368089 29501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:50.374552 29506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.374570 29507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:50.374552 29509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:50.375581 29501 server_base.cc:1034] running on GCE node
I20260928 15:02:50.376034 29501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:50.376632 29501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:50.377892 29501 hybrid_clock.cc:648] HybridClock initialized: now 1790607770377830 us; error 92 us; skew 500 ppm
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:50.382104 29501 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:50.382736 29501 webserver.cc:460] Webserver started at http://127.25.123.3:42957/ using document root <none> and password file <none>
I20260928 15:02:50.382987 29501 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:50.383054 29501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:50.383199 29501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:50.384478 29501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "9a11687b6ea444a4a0d793806949a10c"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-qnz0"
server_key: "73f9f89dd77bf75de3e6077de92292d1"
server_key_iv: "5807ac190786be5b20c78fd47276f89a"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.384898 29501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "9a11687b6ea444a4a0d793806949a10c"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-qnz0"
server_key: "73f9f89dd77bf75de3e6077de92292d1"
server_key_iv: "5807ac190786be5b20c78fd47276f89a"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.386579 29501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:02:50.387588 29516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.387878 29501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:50.388015 29501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "9a11687b6ea444a4a0d793806949a10c"
format_stamp: "Formatted at 2026-09-28 15:02:50 on dist-test-slave-qnz0"
server_key: "73f9f89dd77bf75de3e6077de92292d1"
server_key_iv: "5807ac190786be5b20c78fd47276f89a"
server_key_version: "encryptionkey@0"
I20260928 15:02:50.388103 29501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:50.403465 29501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:50.407270 29501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:50.407632 29501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:50.408054 29501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:50.408504 29501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:50.408561 29501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.408609 29501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:50.408635 29501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:50.415973 29501 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:45001
I20260928 15:02:50.416079 29630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:45001 every 8 connection(s)
I20260928 15:02:50.416635 29501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:50.426692 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29501
I20260928 15:02:50.426812 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:50.427042 26092 external_mini_cluster.cc:1428] Setting key 59d3d2b7fd51dd77c9cc2d57c308b8fb
I20260928 15:02:50.429675 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.418000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:52735 (local address 127.25.123.62:35503)
0928 15:02:50.418055 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:50.418058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:50.418106 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:02:50.418728 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:50.418729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:50.418741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:50.418813 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:50.419854 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.420383 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.421528 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.421788 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.421866 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.425107 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:50.425110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:50.425110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:50.425125 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:50.427182 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.428192 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.428194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.428194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.428273 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.428752 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.428754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.428754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.428902 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:50.428974 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:50.429489 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 15:02:50.429573 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 15:02:50.429705 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.417668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35503 (local address 127.25.123.3:52735)
0928 15:02:50.417955 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:50.418013 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:50.418615 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:50.418891 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:50.418896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:50.419524 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:50.419754 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:50.419760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.420537 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.420538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:50.421421 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:50.421424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.421875 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.421877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:50.422012 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.422780 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:50.422789 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:50.424969 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:50.427310 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:50.427324 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:50.427327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:50.428029 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:50.428463 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:50.428465 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:50.428465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:50.428665 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:50.428999 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:50.429007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:50.429169 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 15:02:50.429296 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:02:50.429491 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260928 15:02:50.430317 29631 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35503
I20260928 15:02:50.430429 29631 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:50.430614 29631 heartbeater.cc:511] Master 127.25.123.62:35503 requested a full tablet report, sending...
I20260928 15:02:50.431097 29167 ts_manager.cc:194] Registered new tserver with Master: 9a11687b6ea444a4a0d793806949a10c (127.25.123.3:45001)
I20260928 15:02:50.431639 29167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:52735
I20260928 15:02:50.440101 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607770, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:50.469058 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:50.458556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.25.123.62:35503)
0928 15:02:50.458664 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:50.458666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:50.458712 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:02:50.458780 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:50.458781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:50.458796 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:50.458879 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:50.459611 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.460497 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.461426 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:50.461687 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:50.461775 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:50.465617 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:50.465620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:50.465620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:50.465654 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:50.467435 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.467949 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.467950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.467951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.468012 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:50.468273 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:50.468275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:50.468275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:50.468557 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:50.468659 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:50.468906 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:02:50.468984 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:02:50.469398 29167 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54548
I20260928 15:02:50.470295 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29230
I20260928 15:02:50.476459 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29366
I20260928 15:02:50.482625 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29501
I20260928 15:02:50.487669 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29135
2026-09-28T15:02:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2579 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29643](info): set up 2 sockets
Sep 28 15:02:50 dist-test-slave-qnz0 krb5kdc[29643](info): commencing operation
krb5kdc: starting...
W20260928 15:02:51.736056 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:51 dist-test-slave-qnz0 krb5kdc[29643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:51.905369 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:36133
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:52.021765 29656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:52.022195 29656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:52.022253 29656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:52.025920 29656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:52.026124 29656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:52.026157 29656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:52.026182 29656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:52.026206 29656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:52.030858 29656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:36133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:52.031322 29656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:52.031757 29656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:52.039780 29662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:52.039834 29661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:52.039924 29664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:52.040360 29656 server_base.cc:1034] running on GCE node
I20260928 15:02:52.040678 29656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:52.041103 29656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:52.042325 29656 hybrid_clock.cc:648] HybridClock initialized: now 1790607772042281 us; error 63 us; skew 500 ppm
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:52.047375 29656 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:52.048590 29656 webserver.cc:460] Webserver started at http://127.25.123.62:33481/ using document root <none> and password file <none>
I20260928 15:02:52.048955 29656 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:52.049041 29656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:52.049212 29656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:52.051712 29656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "a8a2e84fe0d548c085ba46199df471fe"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "a7e65c87e749ae36fec4cad5cdfd65f1"
server_key_iv: "76dc440a0da1e83252a8f0f5d04b3649"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.052232 29656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "a8a2e84fe0d548c085ba46199df471fe"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "a7e65c87e749ae36fec4cad5cdfd65f1"
server_key_iv: "76dc440a0da1e83252a8f0f5d04b3649"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.055349 29656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:02:52.056809 29671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.057215 29656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:52.057368 29656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "a8a2e84fe0d548c085ba46199df471fe"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "a7e65c87e749ae36fec4cad5cdfd65f1"
server_key_iv: "76dc440a0da1e83252a8f0f5d04b3649"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.057469 29656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:52.101765 29656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:52.106204 29656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:52.106459 29656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:52.115301 29656 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:36133
I20260928 15:02:52.115443 29724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:36133 every 8 connection(s)
I20260928 15:02:52.116060 29656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:52.116959 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:52.119769 29725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe: Bootstrap starting.
I20260928 15:02:52.120541 29725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:52.120875 29725 log.cc:824] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:52.122130 29725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe: No bootstrap required, opened a new log
I20260928 15:02:52.123909 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29656
I20260928 15:02:52.124083 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:52.124310 26092 external_mini_cluster.cc:1428] Setting key 8dcc76adcd63841cd4eee0ffe7d74fdb
I20260928 15:02:52.126717 29725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a2e84fe0d548c085ba46199df471fe" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36133 } }
I20260928 15:02:52.126933 29725 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:52.126958 29725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a2e84fe0d548c085ba46199df471fe, State: Initialized, Role: FOLLOWER
I20260928 15:02:52.127122 29725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a2e84fe0d548c085ba46199df471fe" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36133 } }
I20260928 15:02:52.127295 29725 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:52.127357 29725 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:52.127476 29725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:52.128810 29725 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a2e84fe0d548c085ba46199df471fe" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36133 } }
I20260928 15:02:52.128949 29725 leader_election.cc:302] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8a2e84fe0d548c085ba46199df471fe; no voters:
I20260928 15:02:52.129174 29725 leader_election.cc:288] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:52.129225 29730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:52.129428 29730 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [term 1 LEADER]: Becoming Leader. State: Replica: a8a2e84fe0d548c085ba46199df471fe, State: Running, Role: LEADER
I20260928 15:02:52.129510 29725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:52.129596 29730 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a2e84fe0d548c085ba46199df471fe" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36133 } }
I20260928 15:02:52.130435 29732 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a2e84fe0d548c085ba46199df471fe. Latest consensus state: current_term: 1 leader_uuid: "a8a2e84fe0d548c085ba46199df471fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a2e84fe0d548c085ba46199df471fe" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36133 } } }
I20260928 15:02:52.130600 29732 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:52.131134 29731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8a2e84fe0d548c085ba46199df471fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a2e84fe0d548c085ba46199df471fe" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36133 } } }
I20260928 15:02:52.131229 29731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:52.132629 29746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:52.132735 29746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:52.132938 29739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:52.133212 29739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:52.135113 29739 catalog_manager.cc:1347] Generated new cluster ID: 3934b1a8503b467fa1f44fc86dadc867
I20260928 15:02:52.135197 29739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:52.140232 29739 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:52.140868 29739 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:52.142663 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.125539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.25.123.62:36133)
0928 15:02:52.125996 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.126006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.126054 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:02:52.126705 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.126721 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.126822 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.127154 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.128279 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.130112 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.131616 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.132811 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.133080 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.137300 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.137307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.137308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.137344 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.140074 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.140703 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.140707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.140708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.140855 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.141090 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.141091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.141092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.141441 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.141663 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.142216 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.142422 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":229,"threads_started":1}
I20260928 15:02:52.146724 29739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a2e84fe0d548c085ba46199df471fe: Generated new TSK 0
I20260928 15:02:52.146958 29739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:52.198985 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36133
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:52.302359 29751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:52.302686 29751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:52.302735 29751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:52.305373 29751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:52.305518 29751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:52.305584 29751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:52.308835 29751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:52.309188 29751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:52.309445 29751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:52.315197 29756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:52.315392 29757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:52.315186 29759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:52.315555 29751 server_base.cc:1034] running on GCE node
I20260928 15:02:52.315784 29751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:52.316463 29751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:52.317768 29751 hybrid_clock.cc:648] HybridClock initialized: now 1790607772317735 us; error 61 us; skew 500 ppm
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:52.323210 29751 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:52.323948 29751 webserver.cc:460] Webserver started at http://127.25.123.1:36033/ using document root <none> and password file <none>
I20260928 15:02:52.324668 29751 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:52.324748 29751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:52.324895 29751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:52.326538 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "3805a82976ed4a8e82b237c1482bf683"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "eb790fbdc97cc64b4f0b67efba1e0b6a"
server_key_iv: "5646621575f62ceb530f81e3dc60c77a"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.327080 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "3805a82976ed4a8e82b237c1482bf683"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "eb790fbdc97cc64b4f0b67efba1e0b6a"
server_key_iv: "5646621575f62ceb530f81e3dc60c77a"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.328877 29751 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:52.330063 29766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.330457 29751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:52.330608 29751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "3805a82976ed4a8e82b237c1482bf683"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "eb790fbdc97cc64b4f0b67efba1e0b6a"
server_key_iv: "5646621575f62ceb530f81e3dc60c77a"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.330711 29751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:52.355146 29751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:52.358649 29751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:52.358915 29751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:52.359278 29751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:52.360212 29751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:52.360282 29751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.360320 29751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:52.360350 29751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.369958 29751 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:35075
I20260928 15:02:52.370069 29880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:35075 every 8 connection(s)
I20260928 15:02:52.370455 29751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:52.376602 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29751
I20260928 15:02:52.376731 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:52.376962 26092 external_mini_cluster.cc:1428] Setting key c1532597e356ec6165214dc590342140
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:52.388479 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.371384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:36133 (local address 127.25.123.1:33397)
0928 15:02:52.371581 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:52.371629 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:52.373192 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:52.373919 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:52.373923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:52.374240 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:52.374459 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:52.374462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.375647 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.375648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:52.376825 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:52.376829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.377354 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.377356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:52.377511 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.378447 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:52.378460 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:52.381506 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:52.384827 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:52.384866 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:52.384872 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:52.386118 (+ 1246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:52.386718 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:52.386720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:52.386721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:52.386985 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:52.387486 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:52.387497 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:52.387734 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 15:02:52.387910 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 15:02:52.388132 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260928 15:02:52.388391 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.371389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:33397 (local address 127.25.123.62:36133)
0928 15:02:52.371717 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.371720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.371732 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:02:52.373456 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.373458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.373657 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.373797 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.374604 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.375513 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.376933 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.377222 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.377317 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.381772 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.381777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.381777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.381791 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.384356 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.386383 (+ 2027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.386386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.386387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.386595 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.387198 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.387200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.387200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.387386 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.387470 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.388172 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.388300 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260928 15:02:52.389412 29881 heartbeater.cc:347] Connected to a master server at 127.25.123.62:36133
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:02:52.389714 29881 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:52.390000 29881 heartbeater.cc:511] Master 127.25.123.62:36133 requested a full tablet report, sending...
I20260928 15:02:52.390636 29688 ts_manager.cc:194] Registered new tserver with Master: 3805a82976ed4a8e82b237c1482bf683 (127.25.123.1:35075)
I20260928 15:02:52.391605 29688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:33397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:52.434441 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36133
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:52.542498 29886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:52.542761 29886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:52.542790 29886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:52.544028 29886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:52.544076 29886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:52.544106 29886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:52.545883 29886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:52.546128 29886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:52.546341 29886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:52.552767 29894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:52.554087 29891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:52.554119 29886 server_base.cc:1034] running on GCE node
W20260928 15:02:52.554112 29892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:52.554819 29886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:52.555474 29886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:52.556715 29886 hybrid_clock.cc:648] HybridClock initialized: now 1790607772556691 us; error 39 us; skew 500 ppm
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:52.559698 29886 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:52.560339 29886 webserver.cc:460] Webserver started at http://127.25.123.2:33623/ using document root <none> and password file <none>
I20260928 15:02:52.560670 29886 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:52.560729 29886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:52.560866 29886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:52.562443 29886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "d406757e4fca4a438d0f04d0d4ba64d6"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "c4d828e076a080d393232595c90db03d"
server_key_iv: "9f258602af9ba1fee2f0aedd25e870a7"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.563207 29886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "d406757e4fca4a438d0f04d0d4ba64d6"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "c4d828e076a080d393232595c90db03d"
server_key_iv: "9f258602af9ba1fee2f0aedd25e870a7"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.565623 29886 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260928 15:02:52.567569 29901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.568116 29886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:52.568222 29886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "d406757e4fca4a438d0f04d0d4ba64d6"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "c4d828e076a080d393232595c90db03d"
server_key_iv: "9f258602af9ba1fee2f0aedd25e870a7"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.568353 29886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:52.587481 29886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:52.590881 29886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:52.591079 29886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:52.591470 29886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:52.592248 29886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:52.592335 29886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.592381 29886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:52.592403 29886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.601504 29886 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:34565
I20260928 15:02:52.601639 30014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:34565 every 8 connection(s)
I20260928 15:02:52.602313 29886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:52.602715 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 29886
I20260928 15:02:52.602830 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:52.603057 26092 external_mini_cluster.cc:1428] Setting key eef202ca5c8aaaf9b9090fbfe3279a17
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:02:52.617138 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.603359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:36129 (local address 127.25.123.62:36133)
0928 15:02:52.603430 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.603433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.603692 (+ 259us) server_negotiation.cc:407] Connection header received
0928 15:02:52.605236 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.605244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.605259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.605339 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.606313 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.607211 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.609382 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.609614 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.609694 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.612869 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.612873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.612874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.612889 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.615157 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.615858 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.615859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.615860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.615923 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.616270 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.616271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.616272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.616494 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.616563 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.616985 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.617066 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 15:02:52.617274 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.603358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:36133 (local address 127.25.123.2:36129)
0928 15:02:52.603541 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:52.603585 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:52.605056 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:52.605434 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:52.605439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:52.605895 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:52.606123 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:52.606129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.607507 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.607508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:52.609209 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:52.609214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.609738 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.609740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:52.609911 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.610681 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:52.610695 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:52.612653 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:52.615382 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:52.615396 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:52.615399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:52.615685 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:52.616010 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:52.616011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:52.616011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:52.616132 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:52.616573 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:52.616580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:52.616800 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 15:02:52.616898 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:02:52.617051 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260928 15:02:52.618214 30015 heartbeater.cc:347] Connected to a master server at 127.25.123.62:36133
I20260928 15:02:52.618331 30015 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:52.618709 30015 heartbeater.cc:511] Master 127.25.123.62:36133 requested a full tablet report, sending...
I20260928 15:02:52.619405 29688 ts_manager.cc:194] Registered new tserver with Master: d406757e4fca4a438d0f04d0d4ba64d6 (127.25.123.2:34565)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
I20260928 15:02:52.619911 29688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:36129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:52.649266 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36133
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:52.748188 30020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:52.748466 30020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:52.748498 30020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:52.749804 30020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:52.749876 30020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:52.749907 30020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:52.752111 30020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:52.752561 30020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:52.752838 30020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:52.761430 30026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:52.761511 30025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:52.761469 30020 server_base.cc:1034] running on GCE node
W20260928 15:02:52.761430 30028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:52.762126 30020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:52.762481 30020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:52.763747 30020 hybrid_clock.cc:648] HybridClock initialized: now 1790607772763736 us; error 34 us; skew 500 ppm
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:52.767470 30020 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:52.768059 30020 webserver.cc:460] Webserver started at http://127.25.123.3:36793/ using document root <none> and password file <none>
I20260928 15:02:52.768332 30020 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:52.768924 30020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:52.769093 30020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:52.770530 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "b5e00abb654640e495dc936c990984a2"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "14c47937f10d65e236d049f30c05b1fd"
server_key_iv: "0c44ef0efd74e901e3fbc524de8b97ba"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.771979 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "b5e00abb654640e495dc936c990984a2"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "14c47937f10d65e236d049f30c05b1fd"
server_key_iv: "0c44ef0efd74e901e3fbc524de8b97ba"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.775771 30020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260928 15:02:52.776962 30036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.777693 30020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:02:52.777803 30020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "b5e00abb654640e495dc936c990984a2"
format_stamp: "Formatted at 2026-09-28 15:02:52 on dist-test-slave-qnz0"
server_key: "14c47937f10d65e236d049f30c05b1fd"
server_key_iv: "0c44ef0efd74e901e3fbc524de8b97ba"
server_key_version: "encryptionkey@0"
I20260928 15:02:52.777956 30020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:52.789670 30020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:52.793628 30020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:52.794104 30020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:52.794541 30020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:52.795006 30020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:52.795049 30020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.795077 30020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:52.795128 30020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:52.803030 30020 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:37763
I20260928 15:02:52.803136 30151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:37763 every 8 connection(s)
I20260928 15:02:52.803514 30020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:52.807102 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30020
I20260928 15:02:52.807349 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:52.807623 26092 external_mini_cluster.cc:1428] Setting key 3eee531ddb274fc81cfa63d9262f9bd7
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:52.836928 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.804791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:42053 (local address 127.25.123.62:36133)
0928 15:02:52.804881 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.804884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.805252 (+ 368us) server_negotiation.cc:407] Connection header received
0928 15:02:52.807202 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.807204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.807239 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.807301 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.808659 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.809231 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.811200 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.811421 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.811511 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.827323 (+ 15812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.827326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.827326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.827337 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.831889 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.833429 (+ 1540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.833432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.833432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.833544 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.836093 (+ 2549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.836095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.836095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.836255 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.836342 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.836708 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.836825 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:02:52.837169 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.804668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:36133 (local address 127.25.123.3:42053)
0928 15:02:52.804934 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:52.805017 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:52.807019 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:52.807396 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:52.807400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:52.808181 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:52.808539 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:52.808547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.809555 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.809558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:52.811034 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:52.811040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.811525 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.811526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:52.811691 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.824651 (+ 12960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:52.824666 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:52.826914 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:52.832452 (+ 5538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:52.832472 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:52.832476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:52.833310 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:52.835805 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:52.835806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:52.835807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:52.835996 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:52.836329 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:52.836337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:52.836552 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 15:02:52.836669 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 15:02:52.836885 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260928 15:02:52.837746 30152 heartbeater.cc:347] Connected to a master server at 127.25.123.62:36133
I20260928 15:02:52.837872 30152 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:52.838073 30152 heartbeater.cc:511] Master 127.25.123.62:36133 requested a full tablet report, sending...
I20260928 15:02:52.838526 29688 ts_manager.cc:194] Registered new tserver with Master: b5e00abb654640e495dc936c990984a2 (127.25.123.3:37763)
I20260928 15:02:52.839020 29688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:42053
I20260928 15:02:52.846977 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:02:52.880096 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.870548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.25.123.1:35075)
0928 15:02:52.870752 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.870761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.870785 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:02:52.870828 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.870831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.870851 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.870984 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.872065 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.873043 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.873939 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.874370 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.874458 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.877033 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.877046 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.877046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.877058 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.878810 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.879220 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.879224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.879225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.879282 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.879483 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.879484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.879484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.879641 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.879808 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.879861 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.879958 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":124,"threads_started":1}
Sep 28 15:02:52 dist-test-slave-qnz0 krb5kdc[29643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:52.893296 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:52.882388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.25.123.62:36133)
0928 15:02:52.882441 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:52.882443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:52.882454 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:02:52.882492 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:52.882493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:52.882663 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:52.882916 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:52.884127 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.885361 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.886521 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:52.886735 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:52.886812 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:52.890092 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:52.890094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:52.890095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:52.890106 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:52.891623 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.892122 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.892124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.892125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.892190 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:52.892526 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:52.892527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:52.892527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:52.892712 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:52.892888 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:52.893061 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 15:02:52.893174 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 15:02:52.893569 29688 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43608
I20260928 15:02:52.894300 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29751
I20260928 15:02:52.902695 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29886
I20260928 15:02:52.912827 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30020
I20260928 15:02:52.921204 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 29656
2026-09-28T15:02:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2433 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:53 dist-test-slave-qnz0 krb5kdc[30164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:53 dist-test-slave-qnz0 krb5kdc[30164](info): set up 2 sockets
Sep 28 15:02:53 dist-test-slave-qnz0 krb5kdc[30164](info): commencing operation
krb5kdc: starting...
W20260928 15:02:54.296069 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:54 dist-test-slave-qnz0 krb5kdc[30164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:54.477902 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:32999
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:54.590564 30177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:54.590853 30177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:54.590888 30177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:54.592511 30177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:54.592605 30177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:54.592622 30177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:54.592635 30177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:54.592649 30177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:54.594645 30177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:54.595324 30177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:54.595682 30177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:54.601684 30183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:54.601673 30182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:54.601675 30185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:54.602568 30177 server_base.cc:1034] running on GCE node
I20260928 15:02:54.603078 30177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:54.603853 30177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:54.605038 30177 hybrid_clock.cc:648] HybridClock initialized: now 1790607774604995 us; error 61 us; skew 500 ppm
Sep 28 15:02:54 dist-test-slave-qnz0 krb5kdc[30164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:54.608150 30177 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:54.608680 30177 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:54.608739 30177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:54.608901 30177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:54.610090 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "b365cad61b6b43c7bc8947a682de3edb"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-qnz0"
server_key: "6b5992dbbf56e5749d7f026fb2d15031"
server_key_iv: "c7e944354e72cd1d862d55dde8ad19ae"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.610634 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "b365cad61b6b43c7bc8947a682de3edb"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-qnz0"
server_key: "6b5992dbbf56e5749d7f026fb2d15031"
server_key_iv: "c7e944354e72cd1d862d55dde8ad19ae"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.614501 30177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 15:02:54.615820 30191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.616664 30177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:54.616772 30177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "b365cad61b6b43c7bc8947a682de3edb"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-qnz0"
server_key: "6b5992dbbf56e5749d7f026fb2d15031"
server_key_iv: "c7e944354e72cd1d862d55dde8ad19ae"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.616959 30177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:54.639264 30177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:54.642490 30177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:54.642632 30177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:54.648310 30177 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:32999
I20260928 15:02:54.648522 30244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:32999 every 8 connection(s)
I20260928 15:02:54.648723 30177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:54.650367 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:54.652976 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb: Bootstrap starting.
I20260928 15:02:54.653643 30245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:54.653959 30245 log.cc:824] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:54.655146 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb: No bootstrap required, opened a new log
I20260928 15:02:54.656704 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30177
I20260928 15:02:54.656843 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:54.657065 26092 external_mini_cluster.cc:1428] Setting key 4173b8f1957ccf5eb755284598fb7a1b
I20260928 15:02:54.658691 30245 raft_consensus.cc:348] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b365cad61b6b43c7bc8947a682de3edb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32999 } }
I20260928 15:02:54.658929 30245 raft_consensus.cc:374] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:54.658972 30245 raft_consensus.cc:729] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b365cad61b6b43c7bc8947a682de3edb, State: Initialized, Role: FOLLOWER
I20260928 15:02:54.659161 30245 consensus_queue.cc:260] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b365cad61b6b43c7bc8947a682de3edb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32999 } }
I20260928 15:02:54.659268 30245 raft_consensus.cc:388] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:54.659312 30245 raft_consensus.cc:482] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:54.659427 30245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:54.660990 30245 raft_consensus.cc:504] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b365cad61b6b43c7bc8947a682de3edb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32999 } }
I20260928 15:02:54.661124 30245 leader_election.cc:302] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b365cad61b6b43c7bc8947a682de3edb; no voters:
I20260928 15:02:54.661371 30245 leader_election.cc:288] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:54.661427 30250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:54.661615 30250 raft_consensus.cc:686] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [term 1 LEADER]: Becoming Leader. State: Replica: b365cad61b6b43c7bc8947a682de3edb, State: Running, Role: LEADER
I20260928 15:02:54.661708 30245 sys_catalog.cc:564] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:54.661789 30250 consensus_queue.cc:237] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b365cad61b6b43c7bc8947a682de3edb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32999 } }
I20260928 15:02:54.662299 30251 sys_catalog.cc:455] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b365cad61b6b43c7bc8947a682de3edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b365cad61b6b43c7bc8947a682de3edb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32999 } } }
I20260928 15:02:54.662387 30251 sys_catalog.cc:458] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:54.662729 30258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:54.663712 30252 sys_catalog.cc:455] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b365cad61b6b43c7bc8947a682de3edb. Latest consensus state: current_term: 1 leader_uuid: "b365cad61b6b43c7bc8947a682de3edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b365cad61b6b43c7bc8947a682de3edb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32999 } } }
I20260928 15:02:54.663898 30252 sys_catalog.cc:458] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:54.664137 30258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:02:54 dist-test-slave-qnz0 krb5kdc[30164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:54.667399 30258 catalog_manager.cc:1347] Generated new cluster ID: 4817f1e21f2045caab4f97d415fb36a2
I20260928 15:02:54.667480 30258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:54.673774 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:54.657916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.25.123.62:32999)
0928 15:02:54.658362 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:54.658372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:54.658415 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:02:54.658726 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:54.658735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:54.658798 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:54.659261 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:54.660503 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.661983 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.663274 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.664012 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.664145 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:54.667581 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:54.667586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:54.667586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:54.667600 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:54.670678 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:54.671320 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:54.671325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:54.671325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:54.671481 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:54.672067 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:54.672069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:54.672069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:54.672576 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:54.672764 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:54.673086 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 15:02:54.673511 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":130,"threads_started":1}
I20260928 15:02:54.691078 30258 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:54.691867 30258 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:54.699731 30258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb: Generated new TSK 0
I20260928 15:02:54.700018 30258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:02:54.745111 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:32999
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:54.869364 30271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:54.869634 30271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:54.869665 30271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:54.871408 30271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:54.871501 30271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:54.871539 30271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:54.878299 30271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:54.878604 30271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:54.878937 30271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:54.889242 30279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:54.889266 30276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:54.889588 30271 server_base.cc:1034] running on GCE node
W20260928 15:02:54.889242 30277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:54.890552 30271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:54.890913 30271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:54.892272 30271 hybrid_clock.cc:648] HybridClock initialized: now 1790607774892207 us; error 87 us; skew 500 ppm
Sep 28 15:02:54 dist-test-slave-qnz0 krb5kdc[30164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:54.901441 30271 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:02:54.901963 30271 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:54.902057 30271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:54.902245 30271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:54.903967 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "5fe9b9184e924d2abd29a42dcd65f016"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-qnz0"
server_key: "06d8de713b375e126a7747add4e3a517"
server_key_iv: "cf43261e0048b4940b1b1ddddcff544a"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.904479 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "5fe9b9184e924d2abd29a42dcd65f016"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-qnz0"
server_key: "06d8de713b375e126a7747add4e3a517"
server_key_iv: "cf43261e0048b4940b1b1ddddcff544a"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.906610 30271 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:02:54.907671 30285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.908114 30271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:54.908353 30271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "5fe9b9184e924d2abd29a42dcd65f016"
format_stamp: "Formatted at 2026-09-28 15:02:54 on dist-test-slave-qnz0"
server_key: "06d8de713b375e126a7747add4e3a517"
server_key_iv: "cf43261e0048b4940b1b1ddddcff544a"
server_key_version: "encryptionkey@0"
I20260928 15:02:54.908514 30271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:54.957546 30271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:54.964692 30271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:54.965628 30271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:54.965950 30271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:54.966421 30271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:54.966501 30271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.966554 30271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:54.966583 30271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:54.982662 30271 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:42049
I20260928 15:02:54.982837 30399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:42049 every 8 connection(s)
I20260928 15:02:54.983103 30271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:54.986148 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30271
I20260928 15:02:54.986285 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:54.986492 26092 external_mini_cluster.cc:1428] Setting key 2cf2f45b111d7438405d6d87fec98f3d
Sep 28 15:02:54 dist-test-slave-qnz0 krb5kdc[30164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607774, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:02:55.007503 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:54.984161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:51621 (local address 127.25.123.62:32999)
0928 15:02:54.984211 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:54.984213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:54.984781 (+ 568us) server_negotiation.cc:407] Connection header received
0928 15:02:54.985032 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:54.985034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:54.985051 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:54.985122 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:54.986641 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.987391 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.989186 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.990357 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.990489 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.001869 (+ 11380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.001872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.001872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.001886 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.005391 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.005988 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.005990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.005990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.006055 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.006421 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.006423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.006423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.006683 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.006770 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.007325 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.007426 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:02:55.008276 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:54.984000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:32999 (local address 127.25.123.1:51621)
0928 15:02:54.984223 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:54.984278 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:54.984905 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:54.985320 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:54.985324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:54.985638 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:54.986464 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:54.986474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.987510 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.987511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:54.988934 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:54.988939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:54.990571 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:54.990572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:54.990767 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:54.991551 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:54.991563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:55.001607 (+ 10044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:55.005524 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.005538 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.005541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.005851 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.006153 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.006155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.006155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.006290 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.006789 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:55.006796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:55.006962 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.007073 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.008072 (+ 999us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":136,"threads_started":1}
I20260928 15:02:55.009668 30400 heartbeater.cc:347] Connected to a master server at 127.25.123.62:32999
I20260928 15:02:55.009785 30400 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:55.009944 30400 heartbeater.cc:511] Master 127.25.123.62:32999 requested a full tablet report, sending...
I20260928 15:02:55.010632 30209 ts_manager.cc:194] Registered new tserver with Master: 5fe9b9184e924d2abd29a42dcd65f016 (127.25.123.1:42049)
I20260928 15:02:55.011447 30209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:51621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:02:55.061430 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:32999
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:55.207027 30405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:55.207305 30405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:55.207340 30405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:55.208830 30405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:55.208897 30405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:55.208932 30405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:55.211620 30405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:55.212715 30405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:55.213423 30405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:55.225492 30413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.225499 30410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.227514 30411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:55.227699 30405 server_base.cc:1034] running on GCE node
I20260928 15:02:55.227993 30405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:55.228349 30405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:55.229609 30405 hybrid_clock.cc:648] HybridClock initialized: now 1790607775229569 us; error 75 us; skew 500 ppm
Sep 28 15:02:55 dist-test-slave-qnz0 krb5kdc[30164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:55.233705 30405 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:02:55.234534 30405 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:55.234635 30405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:55.235921 30405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:55.241158 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "372de71f01774434aa8fe21d003148b7"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "8b305c80343dc3a12ceae80021994a9d"
server_key_iv: "b7683cca6ecdd66e7074e74217d98ad0"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.242329 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "372de71f01774434aa8fe21d003148b7"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "8b305c80343dc3a12ceae80021994a9d"
server_key_iv: "b7683cca6ecdd66e7074e74217d98ad0"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.245122 30405 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 15:02:55.246937 30419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.247426 30405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:55.247565 30405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "372de71f01774434aa8fe21d003148b7"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "8b305c80343dc3a12ceae80021994a9d"
server_key_iv: "b7683cca6ecdd66e7074e74217d98ad0"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.247675 30405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:55.264739 30405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:55.267460 30405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:55.267594 30405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:55.267932 30405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:55.268290 30405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:55.268326 30405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.268350 30405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:55.268363 30405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.276270 30405 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:37805
I20260928 15:02:55.276569 30533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:37805 every 8 connection(s)
I20260928 15:02:55.276655 30405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:55.281584 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30405
I20260928 15:02:55.281703 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:55.281920 26092 external_mini_cluster.cc:1428] Setting key a11a76aa1e17e98b06c0c22a0bb360b7
Sep 28 15:02:55 dist-test-slave-qnz0 krb5kdc[30164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:02:55.291828 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.277754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:35809 (local address 127.25.123.62:32999)
0928 15:02:55.277805 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.277813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.278100 (+ 287us) server_negotiation.cc:407] Connection header received
0928 15:02:55.278605 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.278606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.278618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.278694 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.279746 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.280771 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.282766 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.282994 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.283084 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.287084 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.287090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.287091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.287111 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.289493 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.290456 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.290458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.290459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.290546 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.290935 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.290936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.290937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.291095 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.291182 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.291566 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.291694 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:02:55.292050 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.277659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:32999 (local address 127.25.123.2:35809)
0928 15:02:55.277925 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.277971 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.278484 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.278755 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.278759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.279044 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:55.279562 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.279567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.281248 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.281250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.282639 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.282647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.283099 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.283100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.283258 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.284266 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:55.284287 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:55.286868 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:55.289629 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.289642 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.289646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.290190 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.290631 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.290633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.290633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.290842 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.291184 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:55.291193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:55.291370 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.291550 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.291768 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":191,"threads_started":1}
I20260928 15:02:55.293040 30534 heartbeater.cc:347] Connected to a master server at 127.25.123.62:32999
I20260928 15:02:55.293217 30534 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:55.293442 30534 heartbeater.cc:511] Master 127.25.123.62:32999 requested a full tablet report, sending...
I20260928 15:02:55.294072 30209 ts_manager.cc:194] Registered new tserver with Master: 372de71f01774434aa8fe21d003148b7 (127.25.123.2:37805)
I20260928 15:02:55.294564 30209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:35809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:02:55.349457 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:32999
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:55.468303 30539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:55.468591 30539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:55.468621 30539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:55.470863 30539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:55.470953 30539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:55.470993 30539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:55.473333 30539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:55.473608 30539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:55.473822 30539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:55.479797 30545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.479702 30547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.479815 30544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:55.480051 30539 server_base.cc:1034] running on GCE node
I20260928 15:02:55.480307 30539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:55.480600 30539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:55.481797 30539 hybrid_clock.cc:648] HybridClock initialized: now 1790607775481762 us; error 48 us; skew 500 ppm
Sep 28 15:02:55 dist-test-slave-qnz0 krb5kdc[30164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:55.486277 30539 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:02:55.486733 30539 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:55.486810 30539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:55.486945 30539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:55.488113 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "815959aa1f3d4df3a90a563a565a2ad0"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "27ab3dd3922cd457caf6816b26bda10d"
server_key_iv: "35a3bc5a51dbf0f836d83da645da928b"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.488547 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "815959aa1f3d4df3a90a563a565a2ad0"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "27ab3dd3922cd457caf6816b26bda10d"
server_key_iv: "35a3bc5a51dbf0f836d83da645da928b"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.490583 30539 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:55.491509 30553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.491827 30539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:55.491963 30539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "815959aa1f3d4df3a90a563a565a2ad0"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "27ab3dd3922cd457caf6816b26bda10d"
server_key_iv: "35a3bc5a51dbf0f836d83da645da928b"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.492059 30539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:55.524781 30539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:55.528438 30539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:55.528654 30539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:55.528954 30539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:55.529309 30539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:55.529363 30539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.529402 30539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:55.529445 30539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.539748 30539 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:38377
I20260928 15:02:55.539876 30667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:38377 every 8 connection(s)
I20260928 15:02:55.540633 30539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:55.548264 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30539
I20260928 15:02:55.548502 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:55.548849 26092 external_mini_cluster.cc:1428] Setting key 0d8117f9b806fe7de0dcab410c978b27
Sep 28 15:02:55 dist-test-slave-qnz0 krb5kdc[30164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607775, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:55.558328 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.542426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:33071 (local address 127.25.123.62:32999)
0928 15:02:55.542490 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.542493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.542747 (+ 254us) server_negotiation.cc:407] Connection header received
0928 15:02:55.543593 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.543594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.543610 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.543899 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.546008 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.546621 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.548371 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.548706 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.548795 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.551631 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.551633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.551633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.551643 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.554310 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.555762 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.555764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.555765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.555858 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.556193 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.556195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.556195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.556994 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.557122 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.558033 (+ 911us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.558223 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:02:55.558501 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.542303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:32999 (local address 127.25.123.3:33071)
0928 15:02:55.542580 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.542635 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.543437 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.543999 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.544004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.544376 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:55.545524 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.545533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.546825 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.546826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.548169 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.548177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.548832 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.548833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.548997 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.549657 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:02:55.549667 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:55.551490 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:55.554460 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.554481 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.554484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.555618 (+ 1134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.555943 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:02:55.555943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:02:55.555944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:02:55.556109 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:02:55.557108 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:55.557117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:55.557367 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.558012 (+ 645us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.558226 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
I20260928 15:02:55.559806 30668 heartbeater.cc:347] Connected to a master server at 127.25.123.62:32999
I20260928 15:02:55.560197 30668 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:55.561239 30668 heartbeater.cc:511] Master 127.25.123.62:32999 requested a full tablet report, sending...
I20260928 15:02:55.561916 30209 ts_manager.cc:194] Registered new tserver with Master: 815959aa1f3d4df3a90a563a565a2ad0 (127.25.123.3:38377)
I20260928 15:02:55.562444 30209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:33071
I20260928 15:02:55.574208 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:55.585801 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.575965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.25.123.62:32999)
0928 15:02:55.576053 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.576054 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.576064 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:55.576092 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.576092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.576102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.576165 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.577177 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.577916 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.579164 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.579571 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.579659 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.581807 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.581810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.581810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:55.581823 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.583782 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.584364 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.584367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.584367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.584607 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:55.584999 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:55.585001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:55.585001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:55.585241 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.585363 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.585612 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.585720 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 15:02:55.589191 30209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:55.591847 30209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:55.601418 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.596438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:37805 (local address 127.0.0.1:51712)
0928 15:02:55.596661 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.596712 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.596796 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.597211 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.597214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.597217 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:55.597312 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.597315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.598714 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.598715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.599831 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.599835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.601000 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.601001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.601106 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.601118 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.601220 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.601285 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":43,"spinlock_wait_cycles":768,"thread_start_us":126,"threads_started":1}
I20260928 15:02:55.601661 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.596438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:38377 (local address 127.0.0.1:46976)
0928 15:02:55.596982 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.596992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.597066 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.597388 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.597389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.597391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:55.597423 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.597425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.598728 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.598729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.600024 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.600028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.601003 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.601005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.601413 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.601421 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.601498 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.601555 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":126,"thread_start_us":100,"threads_started":1}
I20260928 15:02:55.601431 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.596781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.25.123.2:37805)
0928 15:02:55.596983 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.596990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.597000 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:55.597046 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.597048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.597066 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.597181 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:55.597401 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.598585 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.599937 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.600795 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.600877 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.600958 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.601239 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.601285 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260928 15:02:55.602128 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.596706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:42049 (local address 127.0.0.1:51332)
0928 15:02:55.597321 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.597332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.597381 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.597582 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.597583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.597585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:55.597638 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.597640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.598569 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.598571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.600040 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.600042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.601890 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.601891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.601964 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.601969 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.602021 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.602066 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":52,"thread_start_us":89,"threads_started":1}
I20260928 15:02:55.602144 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.596627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.25.123.3:38377)
0928 15:02:55.597214 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.597217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.597227 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:02:55.597263 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.597265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.597281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.597496 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:55.597567 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.598619 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.600122 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.600887 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.600955 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.601021 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.601991 (+ 970us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.602051 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":113,"threads_started":1}
I20260928 15:02:55.603116 30467 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb1bc399a404487a744f5153370bdb8 (DEFAULT_TABLE table=test-table [id=b29b1349cc074cb2bc58af1581fdccc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:55.603112 30598 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb1bc399a404487a744f5153370bdb8 (DEFAULT_TABLE table=test-table [id=b29b1349cc074cb2bc58af1581fdccc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:55.603796 30467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb1bc399a404487a744f5153370bdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:55.603796 30598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb1bc399a404487a744f5153370bdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:55.604773 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.596950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.25.123.1:42049)
0928 15:02:55.597102 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.597108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.597440 (+ 332us) server_negotiation.cc:407] Connection header received
0928 15:02:55.597484 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.597486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.597499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.597585 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:55.597699 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.598469 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.600161 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.601580 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.601647 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.601828 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.602409 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.604592 (+ 2183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260928 15:02:55.605999 30333 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb1bc399a404487a744f5153370bdb8 (DEFAULT_TABLE table=test-table [id=b29b1349cc074cb2bc58af1581fdccc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:55.606467 30333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb1bc399a404487a744f5153370bdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:55.608130 30685 tablet_bootstrap.cc:492] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Bootstrap starting.
I20260928 15:02:55.608150 30686 tablet_bootstrap.cc:492] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Bootstrap starting.
I20260928 15:02:55.608891 30685 tablet_bootstrap.cc:654] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:55.608940 30686 tablet_bootstrap.cc:654] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:55.609452 30685 log.cc:824] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:55.609562 30686 log.cc:824] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:55.609826 30687 tablet_bootstrap.cc:492] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016: Bootstrap starting.
I20260928 15:02:55.610347 30685 tablet_bootstrap.cc:492] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: No bootstrap required, opened a new log
I20260928 15:02:55.610498 30685 ts_tablet_manager.cc:1397] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:55.610687 30687 tablet_bootstrap.cc:654] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:55.610965 30686 tablet_bootstrap.cc:492] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: No bootstrap required, opened a new log
I20260928 15:02:55.611255 30686 ts_tablet_manager.cc:1397] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:02:55.611594 30687 log.cc:824] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:55.614642 30685 raft_consensus.cc:348] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.614939 30687 tablet_bootstrap.cc:492] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016: No bootstrap required, opened a new log
I20260928 15:02:55.615028 30685 raft_consensus.cc:374] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:55.615054 30687 ts_tablet_manager.cc:1397] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 15:02:55.615149 30685 raft_consensus.cc:729] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815959aa1f3d4df3a90a563a565a2ad0, State: Initialized, Role: FOLLOWER
I20260928 15:02:55.615337 30685 consensus_queue.cc:260] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.615536 30686 raft_consensus.cc:348] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.615670 30685 ts_tablet_manager.cc:1428] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 15:02:55.615728 30686 raft_consensus.cc:374] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:55.615788 30668 heartbeater.cc:503] Master 127.25.123.62:32999 was elected leader, sending a full tablet report...
I20260928 15:02:55.615821 30686 raft_consensus.cc:729] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372de71f01774434aa8fe21d003148b7, State: Initialized, Role: FOLLOWER
I20260928 15:02:55.615962 30686 consensus_queue.cc:260] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.616295 30686 ts_tablet_manager.cc:1428] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 15:02:55.616412 30534 heartbeater.cc:503] Master 127.25.123.62:32999 was elected leader, sending a full tablet report...
I20260928 15:02:55.617764 30687 raft_consensus.cc:348] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.617903 30687 raft_consensus.cc:374] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:55.617925 30687 raft_consensus.cc:729] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe9b9184e924d2abd29a42dcd65f016, State: Initialized, Role: FOLLOWER
I20260928 15:02:55.618050 30687 consensus_queue.cc:260] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.618487 30687 ts_tablet_manager.cc:1428] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:02:55.618498 30400 heartbeater.cc:503] Master 127.25.123.62:32999 was elected leader, sending a full tablet report...
I20260928 15:02:55.703081 30691 raft_consensus.cc:482] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:55.703244 30691 raft_consensus.cc:504] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.703603 30691 leader_election.cc:288] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372de71f01774434aa8fe21d003148b7 (127.25.123.2:37805), 5fe9b9184e924d2abd29a42dcd65f016 (127.25.123.1:42049)
I20260928 15:02:55.707722 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.704151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:54627 (local address 127.25.123.2:37805)
0928 15:02:55.704225 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.704227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.704310 (+ 83us) server_negotiation.cc:407] Connection header received
0928 15:02:55.704345 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.704346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.704357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.704449 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:55.704657 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.705294 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.706527 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.707034 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.707094 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.707102 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.707613 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.707656 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:02:55.707749 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.704031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:37805 (local address 127.25.123.3:54627)
0928 15:02:55.704128 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.704143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.704216 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.704458 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.704459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.704462 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:55.704547 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.704550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.705398 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.705399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.706373 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.706376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.707159 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.707162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.707297 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.707306 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.707603 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.707669 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:02:55.708194 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1bc399a404487a744f5153370bdb8" candidate_uuid: "815959aa1f3d4df3a90a563a565a2ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372de71f01774434aa8fe21d003148b7" is_pre_election: true
I20260928 15:02:55.708287 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.704258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:60905 (local address 127.25.123.1:42049)
0928 15:02:55.704307 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.704309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.704425 (+ 116us) server_negotiation.cc:407] Connection header received
0928 15:02:55.704493 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.704494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.704513 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.704591 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:02:55.704929 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.705608 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.706966 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.707910 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.707981 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.707992 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.708170 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.708227 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:02:55.708331 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.704160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:42049 (local address 127.25.123.3:60905)
0928 15:02:55.704346 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.704356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:55.704436 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:55.704721 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:55.704722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:55.704725 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:02:55.704797 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.704805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.705768 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.705769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.706779 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:55.706781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.708010 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.708011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:55.708083 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.708088 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:02:55.708139 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 15:02:55.708175 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260928 15:02:55.708446 30487 raft_consensus.cc:2454] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 815959aa1f3d4df3a90a563a565a2ad0 in term 0.
I20260928 15:02:55.708660 30354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1bc399a404487a744f5153370bdb8" candidate_uuid: "815959aa1f3d4df3a90a563a565a2ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe9b9184e924d2abd29a42dcd65f016" is_pre_election: true
I20260928 15:02:55.708745 30555 leader_election.cc:302] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372de71f01774434aa8fe21d003148b7, 815959aa1f3d4df3a90a563a565a2ad0; no voters:
I20260928 15:02:55.708818 30354 raft_consensus.cc:2454] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 815959aa1f3d4df3a90a563a565a2ad0 in term 0.
I20260928 15:02:55.708930 30691 raft_consensus.cc:2781] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:55.709072 30691 raft_consensus.cc:482] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:55.709100 30691 raft_consensus.cc:3037] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:55.710184 30691 raft_consensus.cc:504] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.710446 30691 leader_election.cc:288] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [CANDIDATE]: Term 1 election: Requested vote from peers 372de71f01774434aa8fe21d003148b7 (127.25.123.2:37805), 5fe9b9184e924d2abd29a42dcd65f016 (127.25.123.1:42049)
I20260928 15:02:55.710868 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1bc399a404487a744f5153370bdb8" candidate_uuid: "815959aa1f3d4df3a90a563a565a2ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372de71f01774434aa8fe21d003148b7"
I20260928 15:02:55.711053 30487 raft_consensus.cc:3037] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:55.711047 30354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1bc399a404487a744f5153370bdb8" candidate_uuid: "815959aa1f3d4df3a90a563a565a2ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe9b9184e924d2abd29a42dcd65f016"
I20260928 15:02:55.711256 30354 raft_consensus.cc:3037] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:55.712055 30354 raft_consensus.cc:2454] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 815959aa1f3d4df3a90a563a565a2ad0 in term 1.
I20260928 15:02:55.712069 30487 raft_consensus.cc:2454] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 815959aa1f3d4df3a90a563a565a2ad0 in term 1.
I20260928 15:02:55.712316 30555 leader_election.cc:302] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372de71f01774434aa8fe21d003148b7, 815959aa1f3d4df3a90a563a565a2ad0; no voters:
I20260928 15:02:55.712747 30691 raft_consensus.cc:2781] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:55.713140 30691 raft_consensus.cc:686] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 815959aa1f3d4df3a90a563a565a2ad0, State: Running, Role: LEADER
I20260928 15:02:55.713308 30691 consensus_queue.cc:237] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } }
I20260928 15:02:55.714661 30209 catalog_manager.cc:5495] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 815959aa1f3d4df3a90a563a565a2ad0 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "815959aa1f3d4df3a90a563a565a2ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "815959aa1f3d4df3a90a563a565a2ad0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 38377 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:55.724004 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.721241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.25.123.3:38377)
0928 15:02:55.721376 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.721378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.721397 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:02:55.721439 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.721440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.721463 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.721557 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:02:55.721767 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.722278 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.723100 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:55.723311 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:55.723408 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:55.723605 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:02:55.723726 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:02:55.723827 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.723874 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.723900 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 15:02:55.726766 30487 raft_consensus.cc:1261] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 1 FOLLOWER]: Refusing update from remote peer 815959aa1f3d4df3a90a563a565a2ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:55.726835 30354 raft_consensus.cc:1261] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 1 FOLLOWER]: Refusing update from remote peer 815959aa1f3d4df3a90a563a565a2ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:55.727600 30691 consensus_queue.cc:1035] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fe9b9184e924d2abd29a42dcd65f016" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:55.727960 30695 consensus_queue.cc:1035] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372de71f01774434aa8fe21d003148b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:55.731556 30699 mvcc.cc:205] Tried to move back new op lower bound from 7334329449372278784 to 7334329449323147264. Current Snapshot: MvccSnapshot[applied={T|T < 7334329449372278784}]
I20260928 15:02:55.733479 30209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:55.733594 30209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:55.734579 30209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b365cad61b6b43c7bc8947a682de3edb: Sending DeleteTablet for 3 replicas of tablet 0bb1bc399a404487a744f5153370bdb8
I20260928 15:02:55.734591 30701 mvcc.cc:205] Tried to move back new op lower bound from 7334329449372278784 to 7334329449323147264. Current Snapshot: MvccSnapshot[applied={T|T < 7334329449372278784 or (T in {7334329449372278784})}]
I20260928 15:02:55.734936 30598 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb1bc399a404487a744f5153370bdb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:55 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:46976
I20260928 15:02:55.734984 30467 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb1bc399a404487a744f5153370bdb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:55 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:51712
I20260928 15:02:55.735230 30709 tablet_replica.cc:330] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: stopping tablet replica
I20260928 15:02:55.735252 30333 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb1bc399a404487a744f5153370bdb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:55 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:51332
I20260928 15:02:55.735329 30709 raft_consensus.cc:2229] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:55.735450 30709 raft_consensus.cc:2258] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:55.735524 30710 tablet_replica.cc:330] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016: stopping tablet replica
I20260928 15:02:55.735654 30710 raft_consensus.cc:2229] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:55.735803 30710 raft_consensus.cc:2258] T 0bb1bc399a404487a744f5153370bdb8 P 5fe9b9184e924d2abd29a42dcd65f016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:55.735899 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30271
I20260928 15:02:55.735911 30708 tablet_replica.cc:330] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: stopping tablet replica
I20260928 15:02:55.736004 30708 raft_consensus.cc:2229] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:55.736094 30708 raft_consensus.cc:2258] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:55.736850 30708 ts_tablet_manager.cc:1905] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:55.739033 30708 ts_tablet_manager.cc:1918] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:55.739108 30708 log.cc:1196] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/0bb1bc399a404487a744f5153370bdb8
I20260928 15:02:55.739398 30708 ts_tablet_manager.cc:1939] T 0bb1bc399a404487a744f5153370bdb8 P 815959aa1f3d4df3a90a563a565a2ad0: Deleting consensus metadata
I20260928 15:02:55.740180 30192 catalog_manager.cc:4841] TS 815959aa1f3d4df3a90a563a565a2ad0 (127.25.123.3:38377): tablet 0bb1bc399a404487a744f5153370bdb8 (table test-table [id=b29b1349cc074cb2bc58af1581fdccc0]) successfully deleted
I20260928 15:02:55.740607 30709 ts_tablet_manager.cc:1905] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:55.742956 30709 ts_tablet_manager.cc:1918] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:55.743065 30709 log.cc:1196] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/0bb1bc399a404487a744f5153370bdb8
I20260928 15:02:55.743366 30709 ts_tablet_manager.cc:1939] T 0bb1bc399a404487a744f5153370bdb8 P 372de71f01774434aa8fe21d003148b7: Deleting consensus metadata
I20260928 15:02:55.744907 30193 catalog_manager.cc:4841] TS 372de71f01774434aa8fe21d003148b7 (127.25.123.2:37805): tablet 0bb1bc399a404487a744f5153370bdb8 (table test-table [id=b29b1349cc074cb2bc58af1581fdccc0]) successfully deleted
W20260928 15:02:55.745198 30193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:42049 (error 108)
I20260928 15:02:55.745339 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30405
I20260928 15:02:55.748080 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.747853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:42049 (local address 127.0.0.1:51344)
0928 15:02:55.747938 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:55.748001 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:02:55.748251 30193 catalog_manager.cc:4584] TS 5fe9b9184e924d2abd29a42dcd65f016 (127.25.123.1:42049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bb1bc399a404487a744f5153370bdb8: Network error: Client connection negotiation failed: client connection to 127.25.123.1:42049: connect: Connection refused (error 111)
I20260928 15:02:55.751802 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30539
I20260928 15:02:55.757396 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30177
2026-09-28T15:02:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2836 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T15:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:55Z Disabled control of system clock
I20260928 15:02:55.792382 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39071
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:02:55.936342 30715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:55.936721 30715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:55.936753 30715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:55.938297 30715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:55.938376 30715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:55.938392 30715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:55.938407 30715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:55.938423 30715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:55.940136 30715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:55.940546 30715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:55.940989 30715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:55.949383 30721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.949553 30720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:55.949383 30723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:55.949574 30715 server_base.cc:1034] running on GCE node
I20260928 15:02:55.950367 30715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:55.950739 30715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:55.952019 30715 hybrid_clock.cc:648] HybridClock initialized: now 1790607775951969 us; error 62 us; skew 500 ppm
I20260928 15:02:55.954254 30715 webserver.cc:460] Webserver started at http://127.25.123.62:38345/ using document root <none> and password file <none>
I20260928 15:02:55.954802 30715 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:55.954914 30715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:55.955102 30715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:55.956383 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "5d8c52767b9b400086fc9b9edbe06488"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "f0d898e79524be6d5e8206b184571b9b"
server_key_iv: "7d63aa6a9a8fb27cf92c1322de56192c"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.957132 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "5d8c52767b9b400086fc9b9edbe06488"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "f0d898e79524be6d5e8206b184571b9b"
server_key_iv: "7d63aa6a9a8fb27cf92c1322de56192c"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.959087 30715 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:02:55.960335 30729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:55.960870 30715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:55.961027 30715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "5d8c52767b9b400086fc9b9edbe06488"
format_stamp: "Formatted at 2026-09-28 15:02:55 on dist-test-slave-qnz0"
server_key: "f0d898e79524be6d5e8206b184571b9b"
server_key_iv: "7d63aa6a9a8fb27cf92c1322de56192c"
server_key_version: "encryptionkey@0"
I20260928 15:02:55.961143 30715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:55.967058 30715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:55.967660 30715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:55.968058 30715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:55.975602 30715 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:39071
I20260928 15:02:55.975716 30781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:39071 every 8 connection(s)
I20260928 15:02:55.976575 30715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:55.977559 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:55.983011 30782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488: Bootstrap starting.
I20260928 15:02:55.983716 30782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:55.984005 30782 log.cc:824] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:55.985147 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30715
I20260928 15:02:55.985256 30782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488: No bootstrap required, opened a new log
I20260928 15:02:55.985376 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:55.985608 26092 external_mini_cluster.cc:1428] Setting key daf2b2cdbf0e944774a82c9bae7d31b1
I20260928 15:02:55.987577 30782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c52767b9b400086fc9b9edbe06488" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39071 } }
I20260928 15:02:55.987782 30782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:55.987823 30782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8c52767b9b400086fc9b9edbe06488, State: Initialized, Role: FOLLOWER
I20260928 15:02:55.988027 30782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c52767b9b400086fc9b9edbe06488" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39071 } }
I20260928 15:02:55.988135 30782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:55.988178 30782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:55.988231 30782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:55.989236 30782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c52767b9b400086fc9b9edbe06488" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39071 } }
I20260928 15:02:55.989392 30782 leader_election.cc:302] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d8c52767b9b400086fc9b9edbe06488; no voters:
I20260928 15:02:55.989449 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:55.986327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.25.123.62:39071)
0928 15:02:55.986576 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:55.986584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:55.986612 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:02:55.987614 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:55.987626 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:55.987697 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:55.987916 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:55.988807 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:55.988812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:55.988812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:55.988818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:55.988875 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:55.988893 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:55.988987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:55.989129 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 15:02:55.989201 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
I20260928 15:02:55.989596 30782 leader_election.cc:288] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:55.989908 30782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:55.989993 30787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:55.990166 30787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [term 1 LEADER]: Becoming Leader. State: Replica: 5d8c52767b9b400086fc9b9edbe06488, State: Running, Role: LEADER
I20260928 15:02:55.990339 30787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c52767b9b400086fc9b9edbe06488" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39071 } }
I20260928 15:02:55.990891 30789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8c52767b9b400086fc9b9edbe06488. Latest consensus state: current_term: 1 leader_uuid: "5d8c52767b9b400086fc9b9edbe06488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c52767b9b400086fc9b9edbe06488" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39071 } } }
I20260928 15:02:55.991009 30789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:55.991050 30788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d8c52767b9b400086fc9b9edbe06488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c52767b9b400086fc9b9edbe06488" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39071 } } }
I20260928 15:02:55.991110 30788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:55.992002 30802 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:02:55.992063 30802 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:55.992115 30799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:55.992270 30799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:55.994634 30799 catalog_manager.cc:1347] Generated new cluster ID: 717a96629e654b568b5910a538161dd4
I20260928 15:02:55.994710 30799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:56.027777 30799 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:56.028225 30799 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:56.034403 30799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488: Generated new TSK 0
I20260928 15:02:56.034651 30799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:56.043776 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39071
--builtin_ntp_servers=127.25.123.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:02:56.153131 30806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:56.153466 30806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:56.153517 30806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:56.155109 30806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:56.155202 30806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:56.155321 30806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:02:56.157974 30806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:56.158504 30806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:56.160154 30806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:56.167950 30812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:56.168072 30806 server_base.cc:1034] running on GCE node
W20260928 15:02:56.167979 30811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:56.167943 30814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:56.168581 30806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:56.168919 30806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:56.170194 30806 hybrid_clock.cc:648] HybridClock initialized: now 1790607776170153 us; error 38 us; skew 500 ppm
I20260928 15:02:56.172842 30806 webserver.cc:460] Webserver started at http://127.25.123.1:46685/ using document root <none> and password file <none>
I20260928 15:02:56.173161 30806 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:56.173223 30806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:56.173349 30806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:56.175051 30806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "e79ade7663b04c00bdccb5ae7322004d"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "6d0245a6e80416baa63cd7276dea3992"
server_key_iv: "c3a7e923df34d200c52ddc7380b9c9bc"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.175562 30806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "e79ade7663b04c00bdccb5ae7322004d"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "6d0245a6e80416baa63cd7276dea3992"
server_key_iv: "c3a7e923df34d200c52ddc7380b9c9bc"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.177326 30806 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 15:02:56.178306 30821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.178498 30806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:56.178603 30806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "e79ade7663b04c00bdccb5ae7322004d"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "6d0245a6e80416baa63cd7276dea3992"
server_key_iv: "c3a7e923df34d200c52ddc7380b9c9bc"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.178725 30806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:56.183099 30806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:56.183432 30806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:56.183548 30806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:56.183807 30806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:56.184155 30806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:56.184212 30806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.184255 30806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:56.184283 30806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.193162 30806 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:37867
I20260928 15:02:56.193290 30936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:37867 every 8 connection(s)
I20260928 15:02:56.193722 30806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:02:56.198170 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.195023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:44571 (local address 127.25.123.62:39071)
0928 15:02:56.195076 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.195079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.195277 (+ 198us) server_negotiation.cc:407] Connection header received
0928 15:02:56.195778 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.195779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.195788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.195896 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.197540 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.197541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.197541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.197546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.197564 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.197568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.197626 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.198046 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.198074 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:02:56.198396 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.194873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39071 (local address 127.25.123.1:44571)
0928 15:02:56.195097 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.195142 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.195638 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.196028 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.196032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.196041 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.197326 (+ 1285us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.197326 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.197361 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.197395 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.197757 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.197774 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.197779 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.197931 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.198050 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":118,"threads_started":1}
I20260928 15:02:56.199594 30937 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39071
I20260928 15:02:56.199759 30937 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:56.199944 30937 heartbeater.cc:511] Master 127.25.123.62:39071 requested a full tablet report, sending...
I20260928 15:02:56.200838 30746 ts_manager.cc:194] Registered new tserver with Master: e79ade7663b04c00bdccb5ae7322004d (127.25.123.1:37867)
I20260928 15:02:56.202020 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30806
I20260928 15:02:56.202173 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:02:56.202373 26092 external_mini_cluster.cc:1428] Setting key 47286f8cc22e3c908c16fd0d47c013b8
I20260928 15:02:56.204070 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39071
--builtin_ntp_servers=127.25.123.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:02:56.304728 30940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:56.304987 30940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:56.305013 30940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:56.306805 30940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:56.306903 30940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:56.306939 30940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:02:56.309674 30940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:56.309993 30940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:56.310521 30940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:56.317736 30945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:56.317740 30946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:56.317740 30948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:56.318140 30940 server_base.cc:1034] running on GCE node
I20260928 15:02:56.318465 30940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:56.318781 30940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:56.319989 30940 hybrid_clock.cc:648] HybridClock initialized: now 1790607776319935 us; error 87 us; skew 500 ppm
I20260928 15:02:56.321794 30940 webserver.cc:460] Webserver started at http://127.25.123.2:44683/ using document root <none> and password file <none>
I20260928 15:02:56.322108 30940 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:56.322167 30940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:56.322309 30940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:56.323508 30940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "01c67ff9802e4d788575bf2695226cfc"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "d505af0532e01f187d5d4b52fd54fa36"
server_key_iv: "b45230d1aebe586450daa59ff0a0cf67"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.323868 30940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "01c67ff9802e4d788575bf2695226cfc"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "d505af0532e01f187d5d4b52fd54fa36"
server_key_iv: "b45230d1aebe586450daa59ff0a0cf67"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.325650 30940 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 15:02:56.327067 30954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.327522 30940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:56.327646 30940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "01c67ff9802e4d788575bf2695226cfc"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "d505af0532e01f187d5d4b52fd54fa36"
server_key_iv: "b45230d1aebe586450daa59ff0a0cf67"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.327710 30940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:56.333127 30940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:56.333415 30940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:56.333523 30940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:56.333706 30940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:56.334055 30940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:56.334089 30940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.334136 30940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:56.334164 30940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.341025 30940 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:35815
I20260928 15:02:56.341130 31068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:35815 every 8 connection(s)
I20260928 15:02:56.341472 30940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:02:56.345980 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.342714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:41399 (local address 127.25.123.62:39071)
0928 15:02:56.342813 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.342815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.342924 (+ 109us) server_negotiation.cc:407] Connection header received
0928 15:02:56.343449 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.343451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.343459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.343524 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.345624 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.345626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.345627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.345633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.345654 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.345660 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.345739 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.345884 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.345903 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:02:56.346226 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.342527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39071 (local address 127.25.123.2:41399)
0928 15:02:56.342749 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.342792 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.343270 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.343648 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.343651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.343662 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.345391 (+ 1729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.345392 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.345430 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.345471 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.345770 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.345777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.345781 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.345870 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.345936 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260928 15:02:56.347137 31069 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39071
I20260928 15:02:56.347275 31069 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:56.347473 31069 heartbeater.cc:511] Master 127.25.123.62:39071 requested a full tablet report, sending...
I20260928 15:02:56.348011 30746 ts_manager.cc:194] Registered new tserver with Master: 01c67ff9802e4d788575bf2695226cfc (127.25.123.2:35815)
I20260928 15:02:56.351414 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 30940
I20260928 15:02:56.351577 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:02:56.351939 26092 external_mini_cluster.cc:1428] Setting key ff2f852f18ca353257776178d77ed01c
I20260928 15:02:56.353731 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39071
--builtin_ntp_servers=127.25.123.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:02:56.453351 31072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:56.453619 31072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:56.453646 31072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:56.455737 31072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:56.455824 31072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:56.455865 31072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:02:56.458175 31072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:56.458515 31072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:56.458923 31072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:02:56.466187 31072 server_base.cc:1034] running on GCE node
W20260928 15:02:56.466157 31080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:56.466157 31077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:56.466148 31078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:56.466889 31072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:56.467191 31072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:56.468537 31072 hybrid_clock.cc:648] HybridClock initialized: now 1790607776468469 us; error 75 us; skew 500 ppm
I20260928 15:02:56.470551 31072 webserver.cc:460] Webserver started at http://127.25.123.3:42369/ using document root <none> and password file <none>
I20260928 15:02:56.470921 31072 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:56.471009 31072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:56.471181 31072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:56.472496 31072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "641eb1e52bec42da84a17e4bccde7650"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "0f3e51982241b4b0893bc2aff36e8543"
server_key_iv: "8e4dd92925fa527164c54db5859020cd"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.473057 31072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "641eb1e52bec42da84a17e4bccde7650"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "0f3e51982241b4b0893bc2aff36e8543"
server_key_iv: "8e4dd92925fa527164c54db5859020cd"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.474972 31072 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:02:56.476617 31086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.476938 31072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:02:56.477157 31072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "641eb1e52bec42da84a17e4bccde7650"
format_stamp: "Formatted at 2026-09-28 15:02:56 on dist-test-slave-qnz0"
server_key: "0f3e51982241b4b0893bc2aff36e8543"
server_key_iv: "8e4dd92925fa527164c54db5859020cd"
server_key_version: "encryptionkey@0"
I20260928 15:02:56.477250 31072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:56.481106 31072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:56.481465 31072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:56.481621 31072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:56.481880 31072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:02:56.482244 31072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:02:56.482302 31072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.482347 31072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:02:56.482373 31072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:56.492566 31072 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:36991
I20260928 15:02:56.492622 31200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:36991 every 8 connection(s)
I20260928 15:02:56.493062 31072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:02:56.497431 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.494033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:34637 (local address 127.25.123.62:39071)
0928 15:02:56.494169 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.494171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.494380 (+ 209us) server_negotiation.cc:407] Connection header received
0928 15:02:56.494951 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.494952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.494960 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.495012 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.496857 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.496861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.496861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.496871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.496905 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.496919 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.497004 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.497156 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.497183 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 15:02:56.498080 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.493943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39071 (local address 127.25.123.3:34637)
0928 15:02:56.494234 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.494286 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.494830 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.495086 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.495090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.495101 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.496299 (+ 1198us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.496300 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.496334 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.496368 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.497003 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.497022 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.497028 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.497338 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.497707 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260928 15:02:56.498672 31201 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39071
I20260928 15:02:56.498793 31201 heartbeater.cc:464] Registering TS with master...
I20260928 15:02:56.498927 31201 heartbeater.cc:511] Master 127.25.123.62:39071 requested a full tablet report, sending...
I20260928 15:02:56.499331 30746 ts_manager.cc:194] Registered new tserver with Master: 641eb1e52bec42da84a17e4bccde7650 (127.25.123.3:36991)
I20260928 15:02:56.501386 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31072
I20260928 15:02:56.501649 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:02:56.501943 26092 external_mini_cluster.cc:1428] Setting key 25147bb2086b9e9aa311e885d944af69
I20260928 15:02:56.504376 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:02:56.508711 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.506171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.25.123.62:39071)
0928 15:02:56.506243 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.506246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.506427 (+ 181us) server_negotiation.cc:407] Connection header received
0928 15:02:56.506899 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.506900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.506907 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.506976 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.508243 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.508245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.508245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.508250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.508267 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.508271 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.508328 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.508620 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.508638 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:02:56.511134 30746 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:02:56.513439 30746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:02:56.520005 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.517538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.25.123.3:36991)
0928 15:02:56.517814 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.517820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.518049 (+ 229us) server_negotiation.cc:407] Connection header received
0928 15:02:56.518175 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.518179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.518259 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.518343 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.519539 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.519556 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.519556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.519566 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.519614 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.519652 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.519709 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.519869 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.519890 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":116,"threads_started":1}
I20260928 15:02:56.519989 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.517530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:35815 (local address 127.0.0.1:56064)
0928 15:02:56.518065 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.518076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.518117 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.518336 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.518346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.518356 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.519148 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.519149 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.519175 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.519213 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.519740 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.519741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.519743 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.519848 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.519867 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":93,"threads_started":1}
I20260928 15:02:56.520012 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.517449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36991 (local address 127.0.0.1:42072)
0928 15:02:56.517926 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.517981 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.518083 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.518778 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.518779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.518782 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.519372 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.519373 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.519383 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.519401 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.519722 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.519730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.519734 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.519865 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.519884 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":80,"threads_started":1}
I20260928 15:02:56.520376 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.517721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.25.123.1:37867)
0928 15:02:56.518015 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.518020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.518479 (+ 459us) server_negotiation.cc:407] Connection header received
0928 15:02:56.518580 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.518583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.518593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.518662 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.519678 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.519685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.519686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.519691 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.519725 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.519748 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.519811 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.520269 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.520288 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":159,"threads_started":1}
I20260928 15:02:56.520634 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.517530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:37867 (local address 127.0.0.1:56228)
0928 15:02:56.518404 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.518416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.518504 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.519140 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.519141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.519144 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.519582 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.519582 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.519588 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.519596 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.520184 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.520185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.520186 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.520535 (+ 349us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.520553 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":414,"thread_start_us":61,"threads_started":1}
I20260928 15:02:56.520839 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.517908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.25.123.2:35815)
0928 15:02:56.518151 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.518158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.518172 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:02:56.518207 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.518210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.518224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.518335 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.519366 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.519371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.519372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.519376 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.519413 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.519448 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.519532 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.519884 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.519905 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260928 15:02:56.521435 30870 tablet_service.cc:1467] Processing CreateTablet for tablet a416388f7de041b898cd49f03c7df54b (DEFAULT_TABLE table=test-table [id=8e4f61288e7f479ca9a7a1c0fdafe543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:56.521636 31003 tablet_service.cc:1467] Processing CreateTablet for tablet a416388f7de041b898cd49f03c7df54b (DEFAULT_TABLE table=test-table [id=8e4f61288e7f479ca9a7a1c0fdafe543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:56.521817 30870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a416388f7de041b898cd49f03c7df54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:56.521641 31135 tablet_service.cc:1467] Processing CreateTablet for tablet a416388f7de041b898cd49f03c7df54b (DEFAULT_TABLE table=test-table [id=8e4f61288e7f479ca9a7a1c0fdafe543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:02:56.522051 31003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a416388f7de041b898cd49f03c7df54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:56.522078 31135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a416388f7de041b898cd49f03c7df54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:56.526012 31216 tablet_bootstrap.cc:492] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Bootstrap starting.
I20260928 15:02:56.526412 31215 tablet_bootstrap.cc:492] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Bootstrap starting.
I20260928 15:02:56.526432 31217 tablet_bootstrap.cc:492] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d: Bootstrap starting.
I20260928 15:02:56.527207 31217 tablet_bootstrap.cc:654] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:56.527207 31215 tablet_bootstrap.cc:654] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:56.527575 31217 log.cc:824] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:56.527575 31215 log.cc:824] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:56.528574 31216 tablet_bootstrap.cc:654] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:56.528752 31217 tablet_bootstrap.cc:492] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d: No bootstrap required, opened a new log
I20260928 15:02:56.528829 31217 ts_tablet_manager.cc:1397] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:56.528954 31216 log.cc:824] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:56.528554 31215 tablet_bootstrap.cc:492] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: No bootstrap required, opened a new log
I20260928 15:02:56.529058 31215 ts_tablet_manager.cc:1397] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:02:56.529989 31216 tablet_bootstrap.cc:492] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: No bootstrap required, opened a new log
I20260928 15:02:56.530063 31216 ts_tablet_manager.cc:1397] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 15:02:56.531443 31215 raft_consensus.cc:348] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.531638 31215 raft_consensus.cc:374] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:56.531687 31215 raft_consensus.cc:729] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c67ff9802e4d788575bf2695226cfc, State: Initialized, Role: FOLLOWER
I20260928 15:02:56.531683 31217 raft_consensus.cc:348] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.531857 31217 raft_consensus.cc:374] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:56.531849 31215 consensus_queue.cc:260] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.531911 31217 raft_consensus.cc:729] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79ade7663b04c00bdccb5ae7322004d, State: Initialized, Role: FOLLOWER
I20260928 15:02:56.532087 31217 consensus_queue.cc:260] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.532723 31217 ts_tablet_manager.cc:1428] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 15:02:56.532787 31215 ts_tablet_manager.cc:1428] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260928 15:02:56.532919 30937 heartbeater.cc:503] Master 127.25.123.62:39071 was elected leader, sending a full tablet report...
I20260928 15:02:56.533061 31069 heartbeater.cc:503] Master 127.25.123.62:39071 was elected leader, sending a full tablet report...
I20260928 15:02:56.533741 31216 raft_consensus.cc:348] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.533923 31216 raft_consensus.cc:374] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:56.533962 31216 raft_consensus.cc:729] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641eb1e52bec42da84a17e4bccde7650, State: Initialized, Role: FOLLOWER
I20260928 15:02:56.534111 31216 consensus_queue.cc:260] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.534536 31216 ts_tablet_manager.cc:1428] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260928 15:02:56.534802 31201 heartbeater.cc:503] Master 127.25.123.62:39071 was elected leader, sending a full tablet report...
W20260928 15:02:56.592550 31070 tablet.cc:2369] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:56.694999 30938 tablet.cc:2369] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:02:56.744059 31202 tablet.cc:2369] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:02:56.805320 31221 raft_consensus.cc:482] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:02:56.805500 31221 raft_consensus.cc:504] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.806010 31221 leader_election.cc:288] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 641eb1e52bec42da84a17e4bccde7650 (127.25.123.3:36991), e79ade7663b04c00bdccb5ae7322004d (127.25.123.1:37867)
I20260928 15:02:56.807752 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.806307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:37867 (local address 127.25.123.2:59803)
0928 15:02:56.806416 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.806431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.806473 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.806678 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.806679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.806683 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.807401 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.807402 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.807410 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.807424 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.807631 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.807632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.807633 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.807680 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.807702 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 15:02:56.807842 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.806400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:59803 (local address 127.25.123.1:37867)
0928 15:02:56.806448 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.806450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.806465 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:02:56.806541 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.806541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.806546 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.806647 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.807518 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.807520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.807520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.807526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.807545 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.807550 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.807611 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.807734 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.807752 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:02:56.808030 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.806307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36991 (local address 127.25.123.2:44295)
0928 15:02:56.806516 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.806525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:56.806581 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:56.806781 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:56.806783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:56.806785 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:02:56.807600 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:02:56.807600 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:02:56.807608 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:02:56.807622 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:02:56.807856 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:02:56.807857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:02:56.807858 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 15:02:56.807921 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 15:02:56.807935 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":24,"thread_start_us":122,"threads_started":1}
I20260928 15:02:56.808140 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.806393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:44295 (local address 127.25.123.3:36991)
0928 15:02:56.806448 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.806450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.806622 (+ 172us) server_negotiation.cc:407] Connection header received
0928 15:02:56.806650 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.806651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.806657 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.806710 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.807726 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.807727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.807728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.807733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.807751 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.807755 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.807813 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.808034 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.808052 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 15:02:56.808486 30890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a416388f7de041b898cd49f03c7df54b" candidate_uuid: "01c67ff9802e4d788575bf2695226cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79ade7663b04c00bdccb5ae7322004d" is_pre_election: true
I20260928 15:02:56.808696 30890 raft_consensus.cc:2454] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c67ff9802e4d788575bf2695226cfc in term 0.
I20260928 15:02:56.808648 31155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a416388f7de041b898cd49f03c7df54b" candidate_uuid: "01c67ff9802e4d788575bf2695226cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641eb1e52bec42da84a17e4bccde7650" is_pre_election: true
I20260928 15:02:56.808811 31155 raft_consensus.cc:2454] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c67ff9802e4d788575bf2695226cfc in term 0.
I20260928 15:02:56.808960 30958 leader_election.cc:302] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c67ff9802e4d788575bf2695226cfc, e79ade7663b04c00bdccb5ae7322004d; no voters:
I20260928 15:02:56.809093 31221 raft_consensus.cc:2781] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:02:56.809149 31221 raft_consensus.cc:482] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:02:56.809175 31221 raft_consensus.cc:3037] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:56.809922 31221 raft_consensus.cc:504] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.810073 31221 leader_election.cc:288] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [CANDIDATE]: Term 1 election: Requested vote from peers 641eb1e52bec42da84a17e4bccde7650 (127.25.123.3:36991), e79ade7663b04c00bdccb5ae7322004d (127.25.123.1:37867)
I20260928 15:02:56.810261 30890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a416388f7de041b898cd49f03c7df54b" candidate_uuid: "01c67ff9802e4d788575bf2695226cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79ade7663b04c00bdccb5ae7322004d"
I20260928 15:02:56.810273 31155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a416388f7de041b898cd49f03c7df54b" candidate_uuid: "01c67ff9802e4d788575bf2695226cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641eb1e52bec42da84a17e4bccde7650"
I20260928 15:02:56.810369 30890 raft_consensus.cc:3037] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:56.810406 31155 raft_consensus.cc:3037] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:56.811389 31155 raft_consensus.cc:2454] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c67ff9802e4d788575bf2695226cfc in term 1.
I20260928 15:02:56.811491 30890 raft_consensus.cc:2454] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c67ff9802e4d788575bf2695226cfc in term 1.
I20260928 15:02:56.811677 30955 leader_election.cc:302] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c67ff9802e4d788575bf2695226cfc, 641eb1e52bec42da84a17e4bccde7650; no voters:
I20260928 15:02:56.811851 31221 raft_consensus.cc:2781] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:56.812062 31221 raft_consensus.cc:686] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 1 LEADER]: Becoming Leader. State: Replica: 01c67ff9802e4d788575bf2695226cfc, State: Running, Role: LEADER
I20260928 15:02:56.812197 31221 consensus_queue.cc:237] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } }
I20260928 15:02:56.814043 30745 catalog_manager.cc:5495] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c67ff9802e4d788575bf2695226cfc (127.25.123.2). New cstate: current_term: 1 leader_uuid: "01c67ff9802e4d788575bf2695226cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c67ff9802e4d788575bf2695226cfc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 35815 } health_report { overall_health: HEALTHY } } }
I20260928 15:02:56.883513 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.882102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.25.123.2:35815)
0928 15:02:56.882171 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:56.882173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:56.882180 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:02:56.882218 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:56.882218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:56.882232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:56.882323 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:56.883142 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:56.883143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:56.883144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:56.883153 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:56.883170 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:56.883174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:56.883310 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:56.883413 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 15:02:56.883429 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:02:56.885707 30890 raft_consensus.cc:1261] T a416388f7de041b898cd49f03c7df54b P e79ade7663b04c00bdccb5ae7322004d [term 1 FOLLOWER]: Refusing update from remote peer 01c67ff9802e4d788575bf2695226cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:56.886086 31155 raft_consensus.cc:1261] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 1 FOLLOWER]: Refusing update from remote peer 01c67ff9802e4d788575bf2695226cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:02:56.886111 31221 consensus_queue.cc:1035] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79ade7663b04c00bdccb5ae7322004d" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:56.886425 31221 consensus_queue.cc:1035] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "641eb1e52bec42da84a17e4bccde7650" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:02:56.889376 31229 mvcc.cc:205] Tried to move back new op lower bound from 7334329454119997440 to 7334329453823528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334329454119997440}]
I20260928 15:02:56.891103 31231 mvcc.cc:205] Tried to move back new op lower bound from 7334329454119997440 to 7334329453823528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334329454119997440}]
I20260928 15:02:56.893080 31228 mvcc.cc:205] Tried to move back new op lower bound from 7334329454119997440 to 7334329453823528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334329454119997440 or (T in {7334329454119997440})}]
I20260928 15:02:56.893077 30746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:56.893194 30746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:02:56.894369 30746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d8c52767b9b400086fc9b9edbe06488: Sending DeleteTablet for 3 replicas of tablet a416388f7de041b898cd49f03c7df54b
I20260928 15:02:56.894670 31003 tablet_service.cc:1514] Processing DeleteTablet for tablet a416388f7de041b898cd49f03c7df54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:56 UTC) from {username='slave'} at 127.0.0.1:56064
I20260928 15:02:56.894677 30870 tablet_service.cc:1514] Processing DeleteTablet for tablet a416388f7de041b898cd49f03c7df54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:56 UTC) from {username='slave'} at 127.0.0.1:56228
I20260928 15:02:56.895128 31239 tablet_replica.cc:330] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: stopping tablet replica
I20260928 15:02:56.895200 31135 tablet_service.cc:1514] Processing DeleteTablet for tablet a416388f7de041b898cd49f03c7df54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:56 UTC) from {username='slave'} at 127.0.0.1:42072
I20260928 15:02:56.895328 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30806
I20260928 15:02:56.895632 31134 tablet_service.cc:1514] Processing DeleteTablet for tablet a416388f7de041b898cd49f03c7df54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:02:56 UTC) from {username='slave'} at 127.0.0.1:42072
I20260928 15:02:56.895381 31239 raft_consensus.cc:2229] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:02:56.896584 31239 raft_consensus.cc:2258] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:56.897081 31239 ts_tablet_manager.cc:1905] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:56.898906 31239 ts_tablet_manager.cc:1918] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:56.898989 31239 log.cc:1196] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/a416388f7de041b898cd49f03c7df54b
I20260928 15:02:56.899355 31239 ts_tablet_manager.cc:1939] T a416388f7de041b898cd49f03c7df54b P 01c67ff9802e4d788575bf2695226cfc: Deleting consensus metadata
I20260928 15:02:56.900384 30731 catalog_manager.cc:4841] TS 01c67ff9802e4d788575bf2695226cfc (127.25.123.2:35815): tablet a416388f7de041b898cd49f03c7df54b (table test-table [id=8e4f61288e7f479ca9a7a1c0fdafe543]) successfully deleted
I20260928 15:02:56.900774 31240 tablet_replica.cc:330] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: stopping tablet replica
I20260928 15:02:56.901111 31240 raft_consensus.cc:2229] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:02:56.901332 31240 raft_consensus.cc:2258] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:02:56.902159 31240 ts_tablet_manager.cc:1905] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:02:56.904121 31240 ts_tablet_manager.cc:1918] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:02:56.904225 31240 log.cc:1196] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/a416388f7de041b898cd49f03c7df54b
I20260928 15:02:56.904556 31240 ts_tablet_manager.cc:1939] T a416388f7de041b898cd49f03c7df54b P 641eb1e52bec42da84a17e4bccde7650: Deleting consensus metadata
I20260928 15:02:56.905530 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30940
W20260928 15:02:56.905653 30733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:37867 (error 108)
I20260928 15:02:56.906952 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:56.906760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:37867 (local address 127.0.0.1:56236)
0928 15:02:56.906843 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:56.906902 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:37867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:02:56.907092 30733 catalog_manager.cc:4584] TS e79ade7663b04c00bdccb5ae7322004d (127.25.123.1:37867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a416388f7de041b898cd49f03c7df54b: Network error: Client connection negotiation failed: client connection to 127.25.123.1:37867: connect: Connection refused (error 111)
I20260928 15:02:56.907409 30730 catalog_manager.cc:4841] TS 641eb1e52bec42da84a17e4bccde7650 (127.25.123.3:36991): tablet a416388f7de041b898cd49f03c7df54b (table test-table [id=8e4f61288e7f479ca9a7a1c0fdafe543]) successfully deleted
W20260928 15:02:56.907939 30730 catalog_manager.cc:4808] TS 641eb1e52bec42da84a17e4bccde7650 (127.25.123.3:36991): delete failed for tablet a416388f7de041b898cd49f03c7df54b because the tablet was not found. No further retry: Not found: Tablet not found: a416388f7de041b898cd49f03c7df54b
I20260928 15:02:56.912137 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31072
I20260928 15:02:56.918081 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 30715
2026-09-28T15:02:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1161 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 15:02:56.942871 26092 mini_oidc.cc:152] Starting JWKS server
I20260928 15:02:56.943960 26092 webserver.cc:460] Webserver started at https://127.0.0.1:38389/ using document root <none> and password file <none>
I20260928 15:02:56.944047 26092 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:02:56.944278 26092 webserver.cc:460] Webserver started at http://127.0.0.1:37201/ using document root <none> and password file <none>
2026-09-28T15:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:56Z Disabled control of system clock
I20260928 15:02:56.954382 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45779
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:45779
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38389/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/testchainca.cert with env {}
W20260928 15:02:57.064008 31247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:57.064327 31247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:57.064379 31247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:57.065820 31247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:57.065913 31247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:57.065935 31247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:57.065956 31247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:57.065990 31247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:02:57.066010 31247 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:02:57.066026 31247 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38389/jwks.json
I20260928 15:02:57.068356 31247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:45779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38389/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:57.068758 31247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:57.069027 31247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:57.075044 31252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.075148 31255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.075039 31253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:57.075237 31247 server_base.cc:1034] running on GCE node
I20260928 15:02:57.075649 31247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:57.076007 31247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:57.077263 31247 hybrid_clock.cc:648] HybridClock initialized: now 1790607777077257 us; error 48 us; skew 500 ppm
I20260928 15:02:57.078948 31247 webserver.cc:460] Webserver started at http://127.25.123.62:39913/ using document root <none> and password file <none>
I20260928 15:02:57.079243 31247 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:57.079334 31247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:57.079500 31247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:57.080924 31247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "5add1138c8534e9f9876bb4b0a977cd9"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "82260d7e521893ef9925bc2e2e704ccc"
server_key_iv: "b6daaaec101382270946bb4955021690"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.081327 31247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "5add1138c8534e9f9876bb4b0a977cd9"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "82260d7e521893ef9925bc2e2e704ccc"
server_key_iv: "b6daaaec101382270946bb4955021690"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.082914 31247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:57.084147 31261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:57.084681 31247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:57.084833 31247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "5add1138c8534e9f9876bb4b0a977cd9"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "82260d7e521893ef9925bc2e2e704ccc"
server_key_iv: "b6daaaec101382270946bb4955021690"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.084949 31247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:57.096107 31247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:57.096642 31247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:57.096832 31247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:57.101581 31247 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:45779
I20260928 15:02:57.101639 31314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:45779 every 8 connection(s)
I20260928 15:02:57.102030 31247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:57.103238 31315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:57.106196 31315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9: Bootstrap starting.
I20260928 15:02:57.106932 31315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:57.107282 31315 log.cc:824] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:57.108500 31315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9: No bootstrap required, opened a new log
I20260928 15:02:57.110320 31315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5add1138c8534e9f9876bb4b0a977cd9" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45779 } }
I20260928 15:02:57.110497 31315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:57.110522 31315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5add1138c8534e9f9876bb4b0a977cd9, State: Initialized, Role: FOLLOWER
I20260928 15:02:57.110700 31315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5add1138c8534e9f9876bb4b0a977cd9" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45779 } }
I20260928 15:02:57.110786 31315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:57.110811 31315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:57.110864 31315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:57.111555 31315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5add1138c8534e9f9876bb4b0a977cd9" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45779 } }
I20260928 15:02:57.111579 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31247
I20260928 15:02:57.111658 31315 leader_election.cc:302] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5add1138c8534e9f9876bb4b0a977cd9; no voters:
I20260928 15:02:57.111752 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:57.111878 31315 leader_election.cc:288] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:57.112059 26092 external_mini_cluster.cc:1428] Setting key a80c27547832b9c5b30f9604045a66e6
I20260928 15:02:57.112053 31318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:57.112223 31315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:57.112322 31318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [term 1 LEADER]: Becoming Leader. State: Replica: 5add1138c8534e9f9876bb4b0a977cd9, State: Running, Role: LEADER
I20260928 15:02:57.112632 31318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5add1138c8534e9f9876bb4b0a977cd9" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45779 } }
I20260928 15:02:57.113531 31320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5add1138c8534e9f9876bb4b0a977cd9. Latest consensus state: current_term: 1 leader_uuid: "5add1138c8534e9f9876bb4b0a977cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5add1138c8534e9f9876bb4b0a977cd9" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45779 } } }
I20260928 15:02:57.113638 31320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.113665 31319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5add1138c8534e9f9876bb4b0a977cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5add1138c8534e9f9876bb4b0a977cd9" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45779 } } }
I20260928 15:02:57.113941 31319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.113847 31335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:57.114558 31335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:57.116415 31335 catalog_manager.cc:1347] Generated new cluster ID: 7566a8ee1d0142a3abbd3554548685e0
I20260928 15:02:57.116473 31335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:57.120461 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.112879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.25.123.62:45779)
0928 15:02:57.113157 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.113168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.113445 (+ 277us) server_negotiation.cc:407] Connection header received
0928 15:02:57.114187 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.114196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.114280 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.114454 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:57.114812 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.116007 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.117187 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.118224 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.118465 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.119391 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:57.119397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:57.119397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:57.119698 (+ 301us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:57.119755 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:57.119774 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:57.119861 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:57.119983 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 15:02:57.120100 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260928 15:02:57.121827 31335 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:57.122388 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:57.122720 31335 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:57.142478 31335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5add1138c8534e9f9876bb4b0a977cd9: Generated new TSK 0
I20260928 15:02:57.142741 31335 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:02:57.175112 31345 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 15:02:57.175243 31345 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:02:57.175526 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.157117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.25.123.62:45779)
0928 15:02:57.157194 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.157196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.157382 (+ 186us) server_negotiation.cc:407] Connection header received
0928 15:02:57.157462 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.157464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.157485 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.157641 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:02:57.157961 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.158829 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.160142 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.160790 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.160889 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.161217 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:02:57.175236 (+ 14019us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:02:57.175440 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 15:02:57.175592 31325 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 15:02:57.176265 31344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:57.156995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45779 (local address 127.0.0.1:34178)
0928 15:02:57.157226 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:57.157238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:57.157327 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:57.157677 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:57.157678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:57.157680 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:02:57.157755 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:57.157757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.158965 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.158966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:57.159986 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:57.159989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.160917 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.160918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:57.160994 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.161033 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:02:57.175463 (+ 14430us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:02:57.175602 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.123.62:45779: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":125,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.25.123.62:45779: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 15:02:57.177196 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31247
2026-09-28T15:02:57Z chronyd exiting
I20260928 15:02:57.216825 26092 test_util.cc:175] -----------------------------------------------
I20260928 15:02:57.216924 26092 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790607748072058-26092-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (275 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 15:02:57.217816 26092 mini_oidc.cc:152] Starting JWKS server
I20260928 15:02:57.218595 26092 webserver.cc:460] Webserver started at https://127.0.0.1:39325/ using document root <none> and password file <none>
I20260928 15:02:57.218645 26092 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:02:57.219000 26092 webserver.cc:460] Webserver started at http://127.0.0.1:35823/ using document root <none> and password file <none>
2026-09-28T15:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:57Z Disabled control of system clock
I20260928 15:02:57.229367 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:46519
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:46519
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39325/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/testchainca.cert with env {}
W20260928 15:02:57.329733 31351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:57.330045 31351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:57.330073 31351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:57.331845 31351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:57.331925 31351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:57.331939 31351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:57.331954 31351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:57.331965 31351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:02:57.331979 31351 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:02:57.331991 31351 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39325/jwks.json
I20260928 15:02:57.333942 31351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:46519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39325/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:57.334225 31351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:57.334815 31351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:57.341629 31359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.341632 31356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.341629 31357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:57.342424 31351 server_base.cc:1034] running on GCE node
I20260928 15:02:57.342854 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:57.343427 31351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:57.344686 31351 hybrid_clock.cc:648] HybridClock initialized: now 1790607777344653 us; error 47 us; skew 500 ppm
I20260928 15:02:57.346692 31351 webserver.cc:460] Webserver started at http://127.25.123.62:45307/ using document root <none> and password file <none>
I20260928 15:02:57.347036 31351 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:57.347119 31351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:57.347301 31351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:57.348631 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "710685d4d9fb40afbd69b3922ea88c33"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "dcbfbf6ee6416aa242afe0f3ccced348"
server_key_iv: "4771772ee9e6c83ac36b9df297c4522d"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.349044 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "710685d4d9fb40afbd69b3922ea88c33"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "dcbfbf6ee6416aa242afe0f3ccced348"
server_key_iv: "4771772ee9e6c83ac36b9df297c4522d"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.350811 31351 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:02:57.351963 31365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:57.352553 31351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:57.352795 31351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "710685d4d9fb40afbd69b3922ea88c33"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "dcbfbf6ee6416aa242afe0f3ccced348"
server_key_iv: "4771772ee9e6c83ac36b9df297c4522d"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.352939 31351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:57.368983 31351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:57.369381 31351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:57.369556 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:57.377403 31351 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:46519
I20260928 15:02:57.377439 31418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:46519 every 8 connection(s)
I20260928 15:02:57.377868 31351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:57.379205 31419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:57.382050 31419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33: Bootstrap starting.
I20260928 15:02:57.382799 31419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:57.383122 31419 log.cc:824] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:57.384510 31419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33: No bootstrap required, opened a new log
I20260928 15:02:57.386750 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31351
I20260928 15:02:57.386894 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:57.386731 31419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710685d4d9fb40afbd69b3922ea88c33" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46519 } }
I20260928 15:02:57.387102 26092 external_mini_cluster.cc:1428] Setting key f6959544cc6b40886885cad9e6e4f962
I20260928 15:02:57.387082 31419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:57.387161 31419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710685d4d9fb40afbd69b3922ea88c33, State: Initialized, Role: FOLLOWER
I20260928 15:02:57.387346 31419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710685d4d9fb40afbd69b3922ea88c33" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46519 } }
I20260928 15:02:57.387447 31419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:57.387492 31419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:57.387609 31419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:57.388585 31419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710685d4d9fb40afbd69b3922ea88c33" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46519 } }
I20260928 15:02:57.388717 31419 leader_election.cc:302] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 710685d4d9fb40afbd69b3922ea88c33; no voters:
I20260928 15:02:57.388973 31419 leader_election.cc:288] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:57.389266 31424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:57.389367 31419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:57.389462 31424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [term 1 LEADER]: Becoming Leader. State: Replica: 710685d4d9fb40afbd69b3922ea88c33, State: Running, Role: LEADER
I20260928 15:02:57.389627 31424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710685d4d9fb40afbd69b3922ea88c33" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46519 } }
I20260928 15:02:57.390115 31425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "710685d4d9fb40afbd69b3922ea88c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710685d4d9fb40afbd69b3922ea88c33" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46519 } } }
I20260928 15:02:57.390214 31425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.390368 31425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 710685d4d9fb40afbd69b3922ea88c33. Latest consensus state: current_term: 1 leader_uuid: "710685d4d9fb40afbd69b3922ea88c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710685d4d9fb40afbd69b3922ea88c33" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 46519 } } }
I20260928 15:02:57.390416 31425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.390722 31433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:57.391908 31433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:57.393886 31433 catalog_manager.cc:1347] Generated new cluster ID: e1631c68081c426ab3543b6eabb156f9
I20260928 15:02:57.393956 31433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:57.394639 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.387692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.25.123.62:46519)
0928 15:02:57.387960 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.387970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.388009 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:02:57.388155 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.388170 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.388252 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.388396 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:57.388613 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.390551 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.392209 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.392861 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.393025 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.393897 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:57.393902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:57.393902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:57.394150 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:57.394188 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:57.394200 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:57.394287 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:57.394366 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 15:02:57.394457 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":102,"threads_started":1}
I20260928 15:02:57.397564 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:57.407187 31433 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:57.407797 31433 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:57.417538 31433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 710685d4d9fb40afbd69b3922ea88c33: Generated new TSK 0
I20260928 15:02:57.417800 31433 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:02:57.443778 31449 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 15:02:57.443977 31449 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:02:57.444330 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.427083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.25.123.62:46519)
0928 15:02:57.427132 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.427134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.427201 (+ 67us) server_negotiation.cc:407] Connection header received
0928 15:02:57.427315 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.427315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.427326 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.427395 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:02:57.427675 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.428548 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.430018 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.430390 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.430486 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.430722 (+ 236us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:02:57.444023 (+ 13301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:02:57.444251 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 15:02:57.444443 31423 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 15:02:57.444448 31448 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:57.426973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:46519 (local address 127.0.0.1:43286)
0928 15:02:57.427134 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:57.427147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:57.427223 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:57.427418 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:57.427419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:57.427422 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:02:57.427489 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:57.427491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.428743 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.428745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:57.429842 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:57.429850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.430490 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.430491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:57.430559 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.430567 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:02:57.444207 (+ 13640us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:02:57.444314 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.123.62:46519: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39325/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260928 15:02:57.444814 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31351
2026-09-28T15:02:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (267 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 15:02:57.485652 26092 mini_oidc.cc:152] Starting JWKS server
I20260928 15:02:57.486552 26092 webserver.cc:460] Webserver started at https://127.0.0.1:37287/ using document root <none> and password file <none>
I20260928 15:02:57.486650 26092 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:02:57.486876 26092 webserver.cc:460] Webserver started at http://127.0.0.1:41073/ using document root <none> and password file <none>
2026-09-28T15:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:57Z Disabled control of system clock
I20260928 15:02:57.498363 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:40729
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:40729
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37287/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:02:57.603048 31455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:57.603404 31455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:57.603434 31455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:57.606937 31455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:57.607043 31455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:57.607059 31455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:57.607074 31455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:57.607089 31455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:02:57.607105 31455 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:02:57.607115 31455 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37287/jwks.json
I20260928 15:02:57.609236 31455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:40729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:40729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37287/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:57.609539 31455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:57.609844 31455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:57.616178 31463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.616191 31461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.616515 31460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:57.616657 31455 server_base.cc:1034] running on GCE node
I20260928 15:02:57.617169 31455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:57.617602 31455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:57.619074 31455 hybrid_clock.cc:648] HybridClock initialized: now 1790607777619090 us; error 91 us; skew 500 ppm
I20260928 15:02:57.621487 31455 webserver.cc:460] Webserver started at http://127.25.123.62:35719/ using document root <none> and password file <none>
I20260928 15:02:57.622083 31455 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:57.622190 31455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:57.622419 31455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:57.624084 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "d9858c7e8d5c405ea803d0882b737470"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "0c9d8209541f6e2d56907e79557e03c3"
server_key_iv: "56352bf368bc3622ff4952455e9a69aa"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.624589 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "d9858c7e8d5c405ea803d0882b737470"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "0c9d8209541f6e2d56907e79557e03c3"
server_key_iv: "56352bf368bc3622ff4952455e9a69aa"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.626689 31455 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 15:02:57.628212 31469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:57.628616 31455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:02:57.628806 31455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "d9858c7e8d5c405ea803d0882b737470"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "0c9d8209541f6e2d56907e79557e03c3"
server_key_iv: "56352bf368bc3622ff4952455e9a69aa"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.628928 31455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:57.649737 31455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:57.650223 31455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:57.650393 31455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:57.655591 31455 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:40729
I20260928 15:02:57.655680 31521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:40729 every 8 connection(s)
I20260928 15:02:57.656339 31455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:57.657454 31522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:57.661029 31522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470: Bootstrap starting.
I20260928 15:02:57.661698 31522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:57.662022 31522 log.cc:824] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:57.662945 31522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470: No bootstrap required, opened a new log
I20260928 15:02:57.664871 31522 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9858c7e8d5c405ea803d0882b737470" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40729 } }
I20260928 15:02:57.665040 31522 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:57.665064 31522 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9858c7e8d5c405ea803d0882b737470, State: Initialized, Role: FOLLOWER
I20260928 15:02:57.665235 31522 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9858c7e8d5c405ea803d0882b737470" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40729 } }
I20260928 15:02:57.665305 31522 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:57.665329 31522 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:57.665395 31522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:57.665690 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31455
I20260928 15:02:57.665815 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:57.666139 26092 external_mini_cluster.cc:1428] Setting key 26b7a8237e3544077cba54537f5429e9
I20260928 15:02:57.666172 31522 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9858c7e8d5c405ea803d0882b737470" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40729 } }
I20260928 15:02:57.666288 31522 leader_election.cc:302] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9858c7e8d5c405ea803d0882b737470; no voters:
I20260928 15:02:57.666656 31522 leader_election.cc:288] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:57.666914 31525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:57.667095 31525 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [term 1 LEADER]: Becoming Leader. State: Replica: d9858c7e8d5c405ea803d0882b737470, State: Running, Role: LEADER
I20260928 15:02:57.667191 31525 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9858c7e8d5c405ea803d0882b737470" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40729 } }
I20260928 15:02:57.667217 31522 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:57.667547 31527 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9858c7e8d5c405ea803d0882b737470. Latest consensus state: current_term: 1 leader_uuid: "d9858c7e8d5c405ea803d0882b737470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9858c7e8d5c405ea803d0882b737470" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40729 } } }
I20260928 15:02:57.667716 31527 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.668129 31528 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9858c7e8d5c405ea803d0882b737470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9858c7e8d5c405ea803d0882b737470" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40729 } } }
I20260928 15:02:57.668217 31528 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.668234 31532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:57.668857 31532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:57.671084 31532 catalog_manager.cc:1347] Generated new cluster ID: 1db66b75076540e586df381e3dad95f9
I20260928 15:02:57.671272 31532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:57.678918 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.666985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.25.123.62:40729)
0928 15:02:57.667410 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.667421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.667471 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:02:57.667612 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.667892 (+ 280us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.668020 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.668597 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:57.668740 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.670136 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.675566 (+ 5430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.676934 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.677151 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.678174 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:57.678178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:57.678179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:57.678377 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:57.678414 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:57.678428 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:57.678507 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:57.678585 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:02:57.678684 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
I20260928 15:02:57.680752 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:57.709964 31532 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:57.710557 31532 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:57.720110 31532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9858c7e8d5c405ea803d0882b737470: Generated new TSK 0
I20260928 15:02:57.720389 31532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:57.741314 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.736322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.25.123.62:40729)
0928 15:02:57.736428 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.736431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.736679 (+ 248us) server_negotiation.cc:407] Connection header received
0928 15:02:57.736884 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.736887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.736923 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.737054 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:02:57.737320 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.737915 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.739073 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.739369 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.739446 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.739690 (+ 244us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:02:57.741059 (+ 1369us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37287/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:02:57.741230 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37287/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 15:02:57.741393 31529 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37287/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 15:02:57.741802 31551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:57.736267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:40729 (local address 127.0.0.1:38286)
0928 15:02:57.736567 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:57.736581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:57.736666 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:57.737075 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:57.737076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:57.737078 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:02:57.737148 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:57.737151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.738028 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.738029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:57.738926 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:57.738929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.739471 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.739472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:57.739537 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.739544 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:02:57.741173 (+ 1629us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37287/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:02:57.741305 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.123.62:40729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37287/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":218,"threads_started":1}
I20260928 15:02:57.742259 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31455
E20260928 15:02:57.752936 31552 webserver.cc:533] Webserver: sslize failed:
E20260928 15:02:57.753091 31552 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T15:02:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T15:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:57Z Disabled control of system clock
I20260928 15:02:57.795518 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:34817
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:34817
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/testchainca.cert with env {}
W20260928 15:02:57.905931 31556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:57.906198 31556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:57.906225 31556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:57.907986 31556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:57.908075 31556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:57.908089 31556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:57.908103 31556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:57.908116 31556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:02:57.908129 31556 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:02:57.908141 31556 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 15:02:57.910642 31556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:34817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:34817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:57.911062 31556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:57.911501 31556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:57.917937 31564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.917892 31562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:57.917937 31561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:57.918409 31556 server_base.cc:1034] running on GCE node
I20260928 15:02:57.918668 31556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:57.919078 31556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:57.920282 31556 hybrid_clock.cc:648] HybridClock initialized: now 1790607777920264 us; error 52 us; skew 500 ppm
I20260928 15:02:57.929176 31556 webserver.cc:460] Webserver started at http://127.25.123.62:41475/ using document root <none> and password file <none>
I20260928 15:02:57.929533 31556 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:57.929594 31556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:57.929733 31556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:57.931041 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "f7178900df524e8e9492fae285b8d047"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "5e3be87bff05c909c1842ab1841ea809"
server_key_iv: "23f0a1cb1da74b48fb340bb836067d9f"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.931665 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "f7178900df524e8e9492fae285b8d047"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "5e3be87bff05c909c1842ab1841ea809"
server_key_iv: "23f0a1cb1da74b48fb340bb836067d9f"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.933533 31556 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 15:02:57.934774 31570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:57.935501 31556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:02:57.935606 31556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "f7178900df524e8e9492fae285b8d047"
format_stamp: "Formatted at 2026-09-28 15:02:57 on dist-test-slave-qnz0"
server_key: "5e3be87bff05c909c1842ab1841ea809"
server_key_iv: "23f0a1cb1da74b48fb340bb836067d9f"
server_key_version: "encryptionkey@0"
I20260928 15:02:57.935866 31556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:57.954627 31556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:57.955020 31556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:57.955188 31556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:57.961220 31556 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:34817
I20260928 15:02:57.961270 31623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:34817 every 8 connection(s)
I20260928 15:02:57.961691 31556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:57.962998 31624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:57.964253 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31556
I20260928 15:02:57.964439 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:57.964639 26092 external_mini_cluster.cc:1428] Setting key 7411c251d52fe323ebae009bae348223
I20260928 15:02:57.966403 31624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047: Bootstrap starting.
I20260928 15:02:57.967140 31624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:57.967478 31624 log.cc:824] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:57.968683 31624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047: No bootstrap required, opened a new log
I20260928 15:02:57.972375 31624 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7178900df524e8e9492fae285b8d047" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34817 } }
I20260928 15:02:57.972812 31624 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:57.972868 31624 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7178900df524e8e9492fae285b8d047, State: Initialized, Role: FOLLOWER
I20260928 15:02:57.973119 31624 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7178900df524e8e9492fae285b8d047" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34817 } }
I20260928 15:02:57.973402 31624 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:57.973551 31624 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:57.973434 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:57.965262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.25.123.62:34817)
0928 15:02:57.965484 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:57.965493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:57.965537 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:02:57.966067 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:57.966075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:57.966142 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:57.966275 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:57.966592 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.968252 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.969728 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:57.970285 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:57.970665 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:57.971780 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:57.971786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:57.971786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:02:57.971987 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:57.972032 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:02:57.972048 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:57.972110 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:57.972512 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 15:02:57.972906 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260928 15:02:57.974066 31624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:57.976522 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:02:57.977792 31624 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7178900df524e8e9492fae285b8d047" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34817 } }
I20260928 15:02:57.977962 31624 leader_election.cc:302] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7178900df524e8e9492fae285b8d047; no voters:
I20260928 15:02:57.978235 31624 leader_election.cc:288] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:57.978346 31630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:57.978667 31630 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [term 1 LEADER]: Becoming Leader. State: Replica: f7178900df524e8e9492fae285b8d047, State: Running, Role: LEADER
I20260928 15:02:57.978785 31630 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7178900df524e8e9492fae285b8d047" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34817 } }
I20260928 15:02:57.978828 31624 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:57.979185 31631 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7178900df524e8e9492fae285b8d047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7178900df524e8e9492fae285b8d047" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34817 } } }
I20260928 15:02:57.979189 31632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7178900df524e8e9492fae285b8d047. Latest consensus state: current_term: 1 leader_uuid: "f7178900df524e8e9492fae285b8d047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7178900df524e8e9492fae285b8d047" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34817 } } }
I20260928 15:02:57.979632 31632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.979439 31631 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:57.980151 31634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:57.980836 31634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:02:57.983425 31634 catalog_manager.cc:1347] Generated new cluster ID: 07c7ea9eba414f2fb9e0adbc8f2fb396
I20260928 15:02:57.983510 31634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:58.012279 31634 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:02:58.013473 31634 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:02:58.028283 31634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7178900df524e8e9492fae285b8d047: Generated new TSK 0
I20260928 15:02:58.028620 31634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:02:58.180991 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:58.035777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.25.123.62:34817)
0928 15:02:58.035889 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:58.035892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:58.035903 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:02:58.035981 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:58.035982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:58.035997 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:58.036107 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:02:58.036545 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.037181 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.038128 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.038359 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.038463 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:58.038690 (+ 227us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:02:58.180110 (+141420us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:02:58.180844 (+ 734us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48686: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:02:58.181085 31627 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48686: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 15:02:58.181394 31654 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:02:58.035245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:34817 (local address 127.0.0.1:48686)
0928 15:02:58.035490 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 15:02:58.035668 (+ 178us) client_negotiation.cc:174] Beginning negotiation
0928 15:02:58.035833 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:02:58.036105 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:02:58.036106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:02:58.036109 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:02:58.036308 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:58.036316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.037319 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.037321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:58.037990 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:02:58.037993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:02:58.038472 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:02:58.038474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:02:58.038544 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:58.038552 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:02:58.180914 (+142362us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:02:58.181251 (+ 337us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.123.62:34817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260928 15:02:58.182062 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31556
2026-09-28T15:02:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (420 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (134 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:02:58 dist-test-slave-qnz0 krb5kdc[31666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:02:58 dist-test-slave-qnz0 krb5kdc[31666](info): set up 2 sockets
Sep 28 15:02:58 dist-test-slave-qnz0 krb5kdc[31666](info): commencing operation
krb5kdc: starting...
W20260928 15:02:59.680451 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:02:59 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:02:59.825162 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38395
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:02:59.916582 31679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:02:59.916898 31679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:02:59.916942 31679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:02:59.918349 31679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:02:59.918469 31679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:02:59.918504 31679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:02:59.918536 31679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:02:59.918568 31679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:02:59.920359 31679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:02:59.920760 31679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:02:59.921036 31679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:02:59.927059 31685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:59.927104 31684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:02:59.927392 31687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:02:59.927870 31679 server_base.cc:1034] running on GCE node
I20260928 15:02:59.928144 31679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:02:59.928520 31679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:02:59.929723 31679 hybrid_clock.cc:648] HybridClock initialized: now 1790607779929704 us; error 32 us; skew 500 ppm
Sep 28 15:02:59 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607779, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:02:59.933679 31679 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:02:59.934381 31679 webserver.cc:460] Webserver started at http://127.25.123.62:44669/ using document root <none> and password file <none>
I20260928 15:02:59.934696 31679 fs_manager.cc:359] Metadata directory not provided
I20260928 15:02:59.934778 31679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:02:59.934957 31679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:02:59.936354 31679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "3975722afbc74e2ebbf6f92db872ff69"
format_stamp: "Formatted at 2026-09-28 15:02:59 on dist-test-slave-qnz0"
server_key: "ad3f4303a7e1127ac1a2764e1b091141"
server_key_iv: "901468b52c1f585c04946ded12353f36"
server_key_version: "encryptionkey@0"
I20260928 15:02:59.937220 31679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "3975722afbc74e2ebbf6f92db872ff69"
format_stamp: "Formatted at 2026-09-28 15:02:59 on dist-test-slave-qnz0"
server_key: "ad3f4303a7e1127ac1a2764e1b091141"
server_key_iv: "901468b52c1f585c04946ded12353f36"
server_key_version: "encryptionkey@0"
I20260928 15:02:59.939960 31679 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260928 15:02:59.941887 31694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:02:59.942278 31679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:02:59.942535 31679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "3975722afbc74e2ebbf6f92db872ff69"
format_stamp: "Formatted at 2026-09-28 15:02:59 on dist-test-slave-qnz0"
server_key: "ad3f4303a7e1127ac1a2764e1b091141"
server_key_iv: "901468b52c1f585c04946ded12353f36"
server_key_version: "encryptionkey@0"
I20260928 15:02:59.942647 31679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:02:59.955485 31679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:02:59.961166 31679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:02:59.961401 31679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:02:59.966936 31679 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:38395
I20260928 15:02:59.967403 31679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:02:59.967286 31747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:38395 every 8 connection(s)
I20260928 15:02:59.968572 31748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:02:59.971817 31748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69: Bootstrap starting.
I20260928 15:02:59.972119 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31679
I20260928 15:02:59.972235 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:02:59.972457 26092 external_mini_cluster.cc:1428] Setting key 871569298dcb3850eb885c6431233b6b
I20260928 15:02:59.973152 31748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69: Neither blocks nor log segments found. Creating new log.
I20260928 15:02:59.973573 31748 log.cc:824] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69: Log is configured to *not* fsync() on all Append() calls
I20260928 15:02:59.974766 31748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69: No bootstrap required, opened a new log
I20260928 15:02:59.976666 31748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3975722afbc74e2ebbf6f92db872ff69" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38395 } }
I20260928 15:02:59.976902 31748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:02:59.976959 31748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3975722afbc74e2ebbf6f92db872ff69, State: Initialized, Role: FOLLOWER
I20260928 15:02:59.977151 31748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3975722afbc74e2ebbf6f92db872ff69" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38395 } }
I20260928 15:02:59.977257 31748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:02:59.977303 31748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:02:59.977355 31748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:02:59.978762 31748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3975722afbc74e2ebbf6f92db872ff69" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38395 } }
I20260928 15:02:59.978930 31748 leader_election.cc:302] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3975722afbc74e2ebbf6f92db872ff69; no voters:
I20260928 15:02:59.979650 31748 leader_election.cc:288] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:02:59.979667 31753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:02:59.980033 31753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [term 1 LEADER]: Becoming Leader. State: Replica: 3975722afbc74e2ebbf6f92db872ff69, State: Running, Role: LEADER
I20260928 15:02:59.980242 31748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:02:59.980214 31753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3975722afbc74e2ebbf6f92db872ff69" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38395 } }
I20260928 15:02:59.982744 31755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3975722afbc74e2ebbf6f92db872ff69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3975722afbc74e2ebbf6f92db872ff69" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38395 } } }
I20260928 15:02:59.983011 31755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [sys.catalog]: This master's current role is: LEADER
I20260928 15:02:59.983206 31754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3975722afbc74e2ebbf6f92db872ff69. Latest consensus state: current_term: 1 leader_uuid: "3975722afbc74e2ebbf6f92db872ff69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3975722afbc74e2ebbf6f92db872ff69" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38395 } } }
W20260928 15:02:59.983458 31768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 15:02:59.983534 31754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69 [sys.catalog]: This master's current role is: LEADER
W20260928 15:02:59.983589 31768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:02:59.983702 31769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:02:59.983896 31769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:02:59 dist-test-slave-qnz0 krb5kdc[31666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:02:59.986930 31769 catalog_manager.cc:1347] Generated new cluster ID: e8cc678bb37141eda2257c0d4d950212
I20260928 15:02:59.986999 31769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:02:59.993229 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:02:59.973353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.25.123.62:38395)
0928 15:02:59.973764 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 15:02:59.973808 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0928 15:02:59.973968 (+ 160us) server_negotiation.cc:407] Connection header received
0928 15:02:59.975041 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:02:59.975050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:02:59.975114 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:02:59.975313 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:02:59.978017 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.979086 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.981867 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:02:59.982814 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:02:59.982947 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:02:59.987431 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:02:59.987437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:02:59.987437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:02:59.987461 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:02:59.990310 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.990871 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.990876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.990877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.990995 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:02:59.991237 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:02:59.991238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:02:59.991238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:02:59.991532 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:02:59.991719 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:02:59.991821 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 15:02:59.992338 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":121,"threads_started":1}
I20260928 15:03:00.006299 31769 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:00.007326 31769 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:00.020687 31769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3975722afbc74e2ebbf6f92db872ff69: Generated new TSK 0
I20260928 15:03:00.021221 31769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:03:00.071976 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38395
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:00.184212 31774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:00.184494 31774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:00.184523 31774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:00.185998 31774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:00.186074 31774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:00.186110 31774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:00.190862 31774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:00.191272 31774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:00.191687 31774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:00.201915 31781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:00.201915 31779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:00.202245 31783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:00.202540 31774 server_base.cc:1034] running on GCE node
I20260928 15:03:00.202922 31774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:00.203329 31774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:00.204638 31774 hybrid_clock.cc:648] HybridClock initialized: now 1790607780204584 us; error 67 us; skew 500 ppm
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.208787 31774 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:03:00.209456 31774 webserver.cc:460] Webserver started at http://127.25.123.1:36637/ using document root <none> and password file <none>
I20260928 15:03:00.209753 31774 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:00.209844 31774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:00.210026 31774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:00.211231 31774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "e93e08c489484f5b9241f13e0781f682"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "a979bf7681bcdc79cb1b1f1105485d6c"
server_key_iv: "dd8236ea03b29134df9728a370b49750"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.211660 31774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "e93e08c489484f5b9241f13e0781f682"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "a979bf7681bcdc79cb1b1f1105485d6c"
server_key_iv: "dd8236ea03b29134df9728a370b49750"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.213559 31774 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 15:03:00.215332 31790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.215857 31774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:03:00.216007 31774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "e93e08c489484f5b9241f13e0781f682"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "a979bf7681bcdc79cb1b1f1105485d6c"
server_key_iv: "dd8236ea03b29134df9728a370b49750"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.216106 31774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:00.234678 31774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:00.238139 31774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:00.238369 31774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:00.238674 31774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:00.239032 31774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:00.239086 31774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.239220 31774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:00.239311 31774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.249578 31774 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:37631
I20260928 15:03:00.249652 31903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:37631 every 8 connection(s)
I20260928 15:03:00.250041 31774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:00.252295 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31774
I20260928 15:03:00.252425 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:00.252640 26092 external_mini_cluster.cc:1428] Setting key 8353955cab96f653e131353b2f627746
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:03:00.264145 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.250982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:50739 (local address 127.25.123.62:38395)
0928 15:03:00.251096 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.251099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.251305 (+ 206us) server_negotiation.cc:407] Connection header received
0928 15:03:00.252722 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.252725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.252751 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.252895 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.254109 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.254653 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.255766 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.255976 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.256067 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.258944 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:00.258947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:00.258947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:00.258965 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:00.261479 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.262022 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.262024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.262025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.262097 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.262586 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.262588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.262588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.262741 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:00.262835 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:00.263947 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0928 15:03:00.264062 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:03:00.264446 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.250870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38395 (local address 127.25.123.1:50739)
0928 15:03:00.251128 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:00.251192 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:00.252518 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:00.253145 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:00.253285 (+ 140us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:00.253625 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:00.253951 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:00.253960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.254774 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.254775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:00.255643 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:00.255645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.256080 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.256081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:00.256244 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.256978 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:00.256988 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:00.258806 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:00.261620 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:00.261635 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:00.261638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:00.261929 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:00.262208 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:00.262210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:00.262210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:00.262387 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:00.263290 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:00.263307 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:00.263547 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 15:03:00.263877 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 15:03:00.264077 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":123,"threads_started":1}
I20260928 15:03:00.265383 31904 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38395
I20260928 15:03:00.265527 31904 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:00.265811 31904 heartbeater.cc:511] Master 127.25.123.62:38395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
I20260928 15:03:00.267290 31712 ts_manager.cc:194] Registered new tserver with Master: e93e08c489484f5b9241f13e0781f682 (127.25.123.1:37631)
I20260928 15:03:00.268040 31712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:50739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:03:00.294154 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38395
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:00.406595 31909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:00.407058 31909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:00.407117 31909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:00.408859 31909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:00.408919 31909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:00.408960 31909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:00.411880 31909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:00.412542 31909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:00.412945 31909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:00.421797 31918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:00.421794 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:00.422091 31909 server_base.cc:1034] running on GCE node
W20260928 15:03:00.421849 31915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:00.422544 31909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:00.423111 31909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:00.424718 31909 hybrid_clock.cc:648] HybridClock initialized: now 1790607780424686 us; error 51 us; skew 500 ppm
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.429071 31909 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:03:00.429844 31909 webserver.cc:460] Webserver started at http://127.25.123.2:36267/ using document root <none> and password file <none>
I20260928 15:03:00.430128 31909 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:00.430205 31909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:00.430371 31909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:00.431746 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "c25797073b8944f592fd7e5fe3250dd4"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "fa0962c9899b0ee6dec7d4730fe53eb4"
server_key_iv: "d06611943e32c54a98334f1c89f66e7f"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.432197 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "c25797073b8944f592fd7e5fe3250dd4"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "fa0962c9899b0ee6dec7d4730fe53eb4"
server_key_iv: "d06611943e32c54a98334f1c89f66e7f"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.434180 31909 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:03:00.435221 31925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.435812 31909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:03:00.435981 31909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "c25797073b8944f592fd7e5fe3250dd4"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "fa0962c9899b0ee6dec7d4730fe53eb4"
server_key_iv: "d06611943e32c54a98334f1c89f66e7f"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.436092 31909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:00.455614 31909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:00.458238 31909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:00.458438 31909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:00.458701 31909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:00.459035 31909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:00.459090 31909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.459134 31909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:00.459161 31909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.467767 31909 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:34831
I20260928 15:03:00.467916 32038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:34831 every 8 connection(s)
I20260928 15:03:00.468215 31909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:00.474968 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 31909
I20260928 15:03:00.475142 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:00.475378 26092 external_mini_cluster.cc:1428] Setting key d02348e3a3b124ccf4edfe5925cf149e
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:00.483733 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.469093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:60621 (local address 127.25.123.62:38395)
0928 15:03:00.469190 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.469193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.469565 (+ 372us) server_negotiation.cc:407] Connection header received
0928 15:03:00.470358 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.470359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.470371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.470440 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.471386 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.472124 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.473492 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.473800 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.473882 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.477241 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:00.477243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:00.477243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:00.477250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:00.479594 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.481955 (+ 2361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.481958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.481959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.482054 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.482836 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.482837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.482838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.482983 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:00.483091 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:00.483516 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 15:03:00.483628 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:03:00.483973 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.469013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38395 (local address 127.25.123.2:60621)
0928 15:03:00.469214 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:00.469273 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:00.470243 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:00.470637 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:00.470641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:00.470937 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:00.471171 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:00.471175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.472263 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.472316 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:00.473337 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:00.473340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.473935 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.473936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:00.474100 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.474917 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:00.474927 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:00.477121 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:00.479931 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:00.479948 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:00.479950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:00.481759 (+ 1809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:00.482220 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:00.482222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:00.482222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:00.482714 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:00.483123 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:00.483132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:00.483350 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 15:03:00.483475 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 15:03:00.483688 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260928 15:03:00.484524 32039 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38395
I20260928 15:03:00.484652 32039 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:00.484889 32039 heartbeater.cc:511] Master 127.25.123.62:38395 requested a full tablet report, sending...
I20260928 15:03:00.485334 31712 ts_manager.cc:194] Registered new tserver with Master: c25797073b8944f592fd7e5fe3250dd4 (127.25.123.2:34831)
I20260928 15:03:00.485757 31712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:60621
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:03:00.532702 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38395
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:00.668867 32044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:00.669155 32044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:00.669185 32044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:00.670660 32044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:00.670740 32044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:00.670774 32044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:00.674003 32044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:00.674548 32044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:00.674902 32044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:00.682644 32049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:00.682644 32052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:00.684325 32044 server_base.cc:1034] running on GCE node
W20260928 15:03:00.684446 32050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:00.685390 32044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:00.686622 32044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:00.688366 32044 hybrid_clock.cc:648] HybridClock initialized: now 1790607780688259 us; error 91 us; skew 500 ppm
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.692572 32044 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:03:00.693239 32044 webserver.cc:460] Webserver started at http://127.25.123.3:46633/ using document root <none> and password file <none>
I20260928 15:03:00.693528 32044 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:00.693610 32044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:00.693782 32044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:00.695006 32044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "0be93a3487b74c49bc41eb9ffe0fa452"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "8b9c20bf23a094ed7de33616d720917b"
server_key_iv: "f7098810ad352c8d0d0b0689e08da626"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.695433 32044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "0be93a3487b74c49bc41eb9ffe0fa452"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "8b9c20bf23a094ed7de33616d720917b"
server_key_iv: "f7098810ad352c8d0d0b0689e08da626"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.697710 32044 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 15:03:00.699440 32060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.700140 32044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:00.700479 32044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "0be93a3487b74c49bc41eb9ffe0fa452"
format_stamp: "Formatted at 2026-09-28 15:03:00 on dist-test-slave-qnz0"
server_key: "8b9c20bf23a094ed7de33616d720917b"
server_key_iv: "f7098810ad352c8d0d0b0689e08da626"
server_key_version: "encryptionkey@0"
I20260928 15:03:00.700618 32044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:00.718026 32044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:00.723213 32044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:00.723491 32044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:00.723857 32044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:00.724300 32044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:00.724364 32044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.724432 32044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:00.724481 32044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:00.734244 32044 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:35087
I20260928 15:03:00.734799 32044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:00.735586 32173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:35087 every 8 connection(s)
I20260928 15:03:00.743451 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 32044
I20260928 15:03:00.743556 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:00.743716 26092 external_mini_cluster.cc:1428] Setting key a1b60a95098abec757c91c3cfd0abb51
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:00.755517 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.735725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38395 (local address 127.25.123.3:41905)
0928 15:03:00.735902 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:00.735952 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:00.737610 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:00.738128 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:00.738133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:00.738530 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:00.738865 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:00.738872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.740939 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.740943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:00.742382 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:00.742390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.743350 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.743352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:00.743727 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.746147 (+ 2420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:00.746164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:00.748925 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:00.751403 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:00.751416 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:00.751419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:00.751851 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:00.753019 (+ 1168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:00.753021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:00.753021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:00.753887 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:00.754656 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:00.754667 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:00.754975 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 15:03:00.755104 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 15:03:00.755300 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 15:03:00.755470 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.735756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:41905 (local address 127.25.123.62:38395)
0928 15:03:00.735837 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.735840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.736326 (+ 486us) server_negotiation.cc:407] Connection header received
0928 15:03:00.737918 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.737920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.737935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.738020 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.739075 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.739940 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.742570 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.742809 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.742907 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.749137 (+ 6230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:00.749141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:00.749141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:00.749155 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:00.751250 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.752042 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.752043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.752044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.752112 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.754095 (+ 1983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.754097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.754097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.754491 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:00.754597 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:00.755206 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 15:03:00.755306 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 15:03:00.756595 32174 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38395
I20260928 15:03:00.756736 32174 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:00.756981 32174 heartbeater.cc:511] Master 127.25.123.62:38395 requested a full tablet report, sending...
I20260928 15:03:00.757596 31712 ts_manager.cc:194] Registered new tserver with Master: 0be93a3487b74c49bc41eb9ffe0fa452 (127.25.123.3:35087)
I20260928 15:03:00.758513 31712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:41905
I20260928 15:03:00.760324 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/test-admin.keytab.
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.804117 26092 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 15:03:00.808069 32181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:00.806274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.25.123.1:37631)
0928 15:03:00.806424 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.806431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.806514 (+ 83us) server_negotiation.cc:407] Connection header received
0928 15:03:00.806706 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.806724 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.806760 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.806893 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.807487 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48128: BlockingRecv error: recv got EOF from 127.0.0.1:48128 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260928 15:03:00.808185 26092 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.810262 26092 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.811827 26092 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:03:00.826941 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.813413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.25.123.1:37631)
0928 15:03:00.813726 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.813729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.813829 (+ 100us) server_negotiation.cc:407] Connection header received
0928 15:03:00.813938 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.813940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.813961 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.814031 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.814524 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.815317 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.816113 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.816562 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.816691 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.819880 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:00.819893 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:00.819893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:00.819902 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:00.823418 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.824798 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.824804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.824804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.824935 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.825571 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.825572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.825572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.825775 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:00.825938 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:00.826629 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 15:03:00.826793 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
W20260928 15:03:00.830518 32181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:00.829295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.25.123.1:37631)
0928 15:03:00.829366 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.829367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.829375 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:00.829396 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.829397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.829407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.829464 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.830416 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48144: BlockingRecv error: recv got EOF from 127.0.0.1:48144 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 15:03:00.830662 26092 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.832894 26092 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.834656 26092 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:03:00.847345 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.835523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.25.123.1:37631)
0928 15:03:00.835592 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.835595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.835603 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:00.835630 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.835631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.835645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.835705 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.836291 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.836895 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.838126 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.838479 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.838555 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.843795 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:00.843797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:00.843798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:00.843882 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:00.846006 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.846496 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.846497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.846500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.846561 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.846779 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.846780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.846780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.846904 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:00.847048 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:00.847126 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:03:00.847223 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 15:03:00.850879 32181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:00.849669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.25.123.1:37631)
0928 15:03:00.849751 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.849754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.850005 (+ 251us) server_negotiation.cc:407] Connection header received
0928 15:03:00.850125 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.850126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.850138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.850220 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.850812 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48170: BlockingRecv error: recv got EOF from 127.0.0.1:48170 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 15:03:00.851089 26092 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790607748072058-26092-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.853109 26092 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:00.855975 26092 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:03:00 dist-test-slave-qnz0 krb5kdc[31666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:03:00.870288 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:00.857424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.25.123.1:37631)
0928 15:03:00.857522 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:00.857530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:00.857543 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:00.857584 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:00.857585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:00.857597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:00.857695 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:00.859342 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.860660 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.861865 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:00.862092 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:00.862173 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:00.865005 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:00.865008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:00.865008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:00.865017 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:00.867290 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.868489 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.868493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.868494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.868957 (+ 463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:00.869391 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:00.869393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:00.869393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:00.869530 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:00.869929 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:00.870111 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 15:03:00.870209 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:03:00.874485 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31774
I20260928 15:03:00.884001 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31909
I20260928 15:03:00.892611 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 32044
I20260928 15:03:00.902263 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 31679
2026-09-28T15:03:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2467 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:01 dist-test-slave-qnz0 krb5kdc[32194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:01 dist-test-slave-qnz0 krb5kdc[32194](info): set up 2 sockets
Sep 28 15:03:01 dist-test-slave-qnz0 krb5kdc[32194](info): commencing operation
krb5kdc: starting...
W20260928 15:03:02.268255 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.340s user 0.002s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:03:02 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab.
Entry for principal oryx/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab.
I20260928 15:03:02.437536 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39247
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab
--principal=oryx/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:02.544932 32208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:02.545188 32208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:02.545219 32208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:02.546665 32208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:02.546728 32208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:02.546744 32208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:02.546757 32208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:02.546769 32208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:02.548930 32208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab
--principal=oryx/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:02.549294 32208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:02.549602 32208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:02.557039 32214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:02.557260 32208 server_base.cc:1034] running on GCE node
W20260928 15:03:02.557458 32213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:02.557353 32216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:02.558084 32208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:02.558439 32208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:02.559635 32208 hybrid_clock.cc:648] HybridClock initialized: now 1790607782559616 us; error 47 us; skew 500 ppm
Sep 28 15:03:02 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:02.563567 32208 init.cc:377] Logged in from keytab as oryx/127.25.123.62@KRBTEST.COM (short username oryx)
I20260928 15:03:02.564189 32208 webserver.cc:460] Webserver started at http://127.25.123.62:34389/ using document root <none> and password file <none>
I20260928 15:03:02.564520 32208 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:02.564599 32208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:02.564774 32208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:02.565974 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "33d29e49a3384726b7be308bc2c0553f"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-qnz0"
server_key: "568250a5d2a1b2ae647e3d284d5f1a9a"
server_key_iv: "c3bfcfad62ecb1012d5c92453c52af9b"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.566376 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "33d29e49a3384726b7be308bc2c0553f"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-qnz0"
server_key: "568250a5d2a1b2ae647e3d284d5f1a9a"
server_key_iv: "c3bfcfad62ecb1012d5c92453c52af9b"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.568200 32208 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 15:03:02.569296 32223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:02.569626 32208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:02.569835 32208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "33d29e49a3384726b7be308bc2c0553f"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-qnz0"
server_key: "568250a5d2a1b2ae647e3d284d5f1a9a"
server_key_iv: "c3bfcfad62ecb1012d5c92453c52af9b"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.569944 32208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:02.599320 32208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:02.603236 32208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:02.603447 32208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:02.608793 32208 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:39247
I20260928 15:03:02.608981 32275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:39247 every 8 connection(s)
I20260928 15:03:02.609301 32208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:02.610479 32276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:02.613718 32276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: Bootstrap starting.
I20260928 15:03:02.614554 32276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:02.614928 32276 log.cc:824] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:02.616142 32276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: No bootstrap required, opened a new log
I20260928 15:03:02.616549 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 32208
I20260928 15:03:02.616693 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:03:02.617133 26092 external_mini_cluster.cc:1428] Setting key 7ca87a8ff88b98844e541702677530b0
I20260928 15:03:02.620224 32276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } }
I20260928 15:03:02.620492 32276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:02.620538 32276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d29e49a3384726b7be308bc2c0553f, State: Initialized, Role: FOLLOWER
I20260928 15:03:02.620710 32276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } }
I20260928 15:03:02.620819 32276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:02.620870 32276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:02.620944 32276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:02.621812 32276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } }
I20260928 15:03:02.621944 32276 leader_election.cc:302] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33d29e49a3384726b7be308bc2c0553f; no voters:
I20260928 15:03:02.622139 32276 leader_election.cc:288] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:02.622314 32282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:02.622439 32276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:02.622566 32282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 1 LEADER]: Becoming Leader. State: Replica: 33d29e49a3384726b7be308bc2c0553f, State: Running, Role: LEADER
I20260928 15:03:02.622792 32282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } }
I20260928 15:03:02.623265 32281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } }
I20260928 15:03:02.623356 32281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:02.623577 32290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:02.624306 32283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d29e49a3384726b7be308bc2c0553f. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } }
I20260928 15:03:02.624409 32283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:02.624423 32290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:02.626873 32290 catalog_manager.cc:1347] Generated new cluster ID: 51bc2c73f52c4394b43f49f3bf2a9094
I20260928 15:03:02.626932 32290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:03:02 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:02.631052 32290 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:02.631592 32290 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:02.633767 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.617868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.25.123.62:39247)
0928 15:03:02.618304 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.618315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.618356 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:03:02.620145 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.620156 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.620244 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.620456 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.621347 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:02.622964 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:02.624172 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:02.625314 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:02.625405 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:02.628559 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.628564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.628565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:02.628580 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.631318 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:02.631810 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:02.631813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:02.631814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:02.631930 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:02.632140 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:02.632141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:02.632141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:02.632489 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.632665 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.633229 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.633434 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":108,"threads_started":1}
I20260928 15:03:02.636049 32290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: Generated new TSK 0
I20260928 15:03:02.636250 32290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab.
Entry for principal oryx/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab.
I20260928 15:03:02.682322 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39247
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab
--principal=oryx/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:02.792491 32302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:02.792786 32302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:02.792819 32302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:02.794476 32302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:02.794747 32302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:02.794823 32302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:02.797333 32302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab
--principal=oryx/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:02.797736 32302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:02.798390 32302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:02.805625 32307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:02.805712 32310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:02.805635 32308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:02.806049 32302 server_base.cc:1034] running on GCE node
I20260928 15:03:02.806303 32302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:02.806663 32302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:02.807902 32302 hybrid_clock.cc:648] HybridClock initialized: now 1790607782807866 us; error 56 us; skew 500 ppm
Sep 28 15:03:02 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:02.811499 32302 init.cc:377] Logged in from keytab as oryx/127.25.123.1@KRBTEST.COM (short username oryx)
I20260928 15:03:02.812101 32302 webserver.cc:460] Webserver started at http://127.25.123.1:42741/ using document root <none> and password file <none>
I20260928 15:03:02.812457 32302 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:02.812517 32302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:02.813041 32302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:02.814666 32302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "15bf9f5050714a10b3d6b720cc8ae6ad"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-qnz0"
server_key: "71f554bee5efacb4501bd3cbb90b769e"
server_key_iv: "065ee216446bc18febaa767c9d491d3b"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.815171 32302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "15bf9f5050714a10b3d6b720cc8ae6ad"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-qnz0"
server_key: "71f554bee5efacb4501bd3cbb90b769e"
server_key_iv: "065ee216446bc18febaa767c9d491d3b"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.818902 32302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 15:03:02.821282 32317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:02.821553 32302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:02.821669 32302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "15bf9f5050714a10b3d6b720cc8ae6ad"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-qnz0"
server_key: "71f554bee5efacb4501bd3cbb90b769e"
server_key_iv: "065ee216446bc18febaa767c9d491d3b"
server_key_version: "encryptionkey@0"
I20260928 15:03:02.821798 32302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:02.852047 32302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:02.855491 32302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:02.855652 32302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:02.855911 32302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:02.856302 32302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:02.856344 32302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:02.856370 32302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:02.856383 32302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:02.865660 32302 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:33601
I20260928 15:03:02.865767 32432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:33601 every 8 connection(s)
I20260928 15:03:02.866142 32302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:02.870853 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 32302
I20260928 15:03:02.870947 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:02.871255 26092 external_mini_cluster.cc:1428] Setting key 5bdf7e94cfc5869e7a31f9e193215cb4
Sep 28 15:03:02 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:02.880430 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.867139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:42683 (local address 127.25.123.62:39247)
0928 15:03:02.867236 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:02.867238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:02.867544 (+ 306us) server_negotiation.cc:407] Connection header received
0928 15:03:02.868712 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:02.868714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:02.868767 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:02.868832 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:02.870087 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:02.870660 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:02.872518 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:02.872788 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:02.873245 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:02.876884 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:02.876893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:02.876893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:02.876903 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:02.878456 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:02.879120 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:02.879122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:02.879123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:02.879237 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:02.879553 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:02.879554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:02.879555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:02.879714 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:02.879807 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:02.880142 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 15:03:02.880307 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:03:02.880606 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:02.867021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:42683)
0928 15:03:02.867318 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:02.867410 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:02.868597 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:02.869155 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:02.869161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:02.869581 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:02.869872 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:02.869880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:02.870898 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:02.870900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:02.872381 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:02.872433 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:02.873272 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:02.873274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:02.873600 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:02.874462 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:02.874475 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:02.876735 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:02.878571 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:02.878583 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:02.878586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:02.878997 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:02.879330 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:02.879330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:02.879331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:02.879471 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:02.879796 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:02.879804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:02.879988 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 15:03:02.880105 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 15:03:02.880311 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for oryx/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:03:02.881186 32433 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39247
I20260928 15:03:02.881312 32433 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:02.881582 32433 heartbeater.cc:511] Master 127.25.123.62:39247 requested a full tablet report, sending...
I20260928 15:03:02.882493 32240 ts_manager.cc:194] Registered new tserver with Master: 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601)
I20260928 15:03:02.883554 32240 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:42683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab.
Entry for principal oryx/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab.
I20260928 15:03:02.909617 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39247
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab
--principal=oryx/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:03.020480 32438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:03.020753 32438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:03.020783 32438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:03.022746 32438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:03.022822 32438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:03.022858 32438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:03.025032 32438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab
--principal=oryx/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:03.025375 32438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:03.025681 32438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:03.032891 32446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:03.033217 32438 server_base.cc:1034] running on GCE node
W20260928 15:03:03.033310 32443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.033380 32444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:03.033713 32438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:03.034061 32438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:03.035301 32438 hybrid_clock.cc:648] HybridClock initialized: now 1790607783035223 us; error 91 us; skew 500 ppm
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:03.043679 32438 init.cc:377] Logged in from keytab as oryx/127.25.123.2@KRBTEST.COM (short username oryx)
I20260928 15:03:03.044324 32438 webserver.cc:460] Webserver started at http://127.25.123.2:38795/ using document root <none> and password file <none>
I20260928 15:03:03.044596 32438 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:03.044673 32438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:03.044931 32438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:03.046195 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "672eadd32ab8224fdae00557021c75bf"
server_key_iv: "47edebd1c2c490b032e2fe0487c2953e"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.046630 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "672eadd32ab8224fdae00557021c75bf"
server_key_iv: "47edebd1c2c490b032e2fe0487c2953e"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.048751 32438 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:03.049849 32453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.050207 32438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:03.050441 32438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "672eadd32ab8224fdae00557021c75bf"
server_key_iv: "47edebd1c2c490b032e2fe0487c2953e"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.050537 32438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:03.077798 32438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:03.081580 32438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:03.081867 32438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:03.082147 32438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:03.082502 32438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:03.082564 32438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.082612 32438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:03.082643 32438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.092881 32438 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:43599
I20260928 15:03:03.092940 32568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:43599 every 8 connection(s)
I20260928 15:03:03.093397 32438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:03.098114 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 32438
I20260928 15:03:03.098456 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:03.098678 26092 external_mini_cluster.cc:1428] Setting key 4d0487f900920865f0ca2f7d28365f95
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:03.108731 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.094401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:43605 (local address 127.25.123.62:39247)
0928 15:03:03.094567 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.094571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.094816 (+ 245us) server_negotiation.cc:407] Connection header received
0928 15:03:03.095437 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.095438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.095451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.095545 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.096668 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.097563 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.099596 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.099795 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.099879 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.104030 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.104033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.104033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.104043 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.106296 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.107063 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.107064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.107064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.107132 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.107628 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.107630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.107631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.107864 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.107975 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.108530 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.108645 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 15:03:03.108950 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.094310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:43605)
0928 15:03:03.094600 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.094657 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.095316 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.095636 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.095639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.096288 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.096546 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.096551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.097894 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.097895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.099430 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.099434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.099890 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.099891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.100032 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.101239 (+ 1207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.101255 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.103671 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.106463 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.106483 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.106487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.106944 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.107220 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.107221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.107222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.107405 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.107992 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.108000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.108217 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.108414 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.108629 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:03:03.109573 32569 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39247
I20260928 15:03:03.109692 32569 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:03.109923 32569 heartbeater.cc:511] Master 127.25.123.62:39247 requested a full tablet report, sending...
I20260928 15:03:03.110409 32240 ts_manager.cc:194] Registered new tserver with Master: cfc7ef5285b74c54a9ac758bb7b0bebc (127.25.123.2:43599)
I20260928 15:03:03.110948 32240 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:43605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab.
Entry for principal oryx/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab.
I20260928 15:03:03.146283 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39247
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab
--principal=oryx/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:03.257826 32574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:03.258165 32574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:03.258211 32574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:03.260077 32574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:03.260164 32574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:03.260216 32574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:03.262077 32574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab
--principal=oryx/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:03.262404 32574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:03.262662 32574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:03.268596 32579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.268597 32582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.269321 32580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:03.269498 32574 server_base.cc:1034] running on GCE node
I20260928 15:03:03.269760 32574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:03.270047 32574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:03.271304 32574 hybrid_clock.cc:648] HybridClock initialized: now 1790607783271226 us; error 88 us; skew 500 ppm
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:03.275492 32574 init.cc:377] Logged in from keytab as oryx/127.25.123.3@KRBTEST.COM (short username oryx)
I20260928 15:03:03.276033 32574 webserver.cc:460] Webserver started at http://127.25.123.3:43135/ using document root <none> and password file <none>
I20260928 15:03:03.276301 32574 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:03.276353 32574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:03.276505 32574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:03.277675 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "6a83f9be73ec1cb3a4a8045277cf1859"
server_key_iv: "013b74cf116bfbd97f604d537a2bf040"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.278059 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "6a83f9be73ec1cb3a4a8045277cf1859"
server_key_iv: "013b74cf116bfbd97f604d537a2bf040"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.279897 32574 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:03.281215 32589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.281898 32574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:03.282047 32574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "6a83f9be73ec1cb3a4a8045277cf1859"
server_key_iv: "013b74cf116bfbd97f604d537a2bf040"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.282320 32574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:03.294514 32574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:03.299131 32574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:03.299398 32574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:03.299947 32574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:03.300676 32574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:03.300738 32574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.300768 32574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:03.300783 32574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.312708 32574 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:39377
I20260928 15:03:03.312832 32703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:39377 every 8 connection(s)
I20260928 15:03:03.313270 32574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:03.313697 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 32574
I20260928 15:03:03.313823 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:03.314074 26092 external_mini_cluster.cc:1428] Setting key 40a9d39459c636998e822e785de53273
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:03.327292 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.314567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:48085 (local address 127.25.123.62:39247)
0928 15:03:03.314718 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.314720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.314857 (+ 137us) server_negotiation.cc:407] Connection header received
0928 15:03:03.315747 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.315748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.315763 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.315838 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.317151 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.317852 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.319072 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.319318 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.319407 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.322846 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.322849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.322849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.322859 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.324778 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.325474 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.325476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.325477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.325548 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.325995 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.325997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.325997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.326172 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.326471 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.326883 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.327016 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 15:03:03.327695 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.314361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:48085)
0928 15:03:03.314627 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.314735 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.315617 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.315923 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.315927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.316787 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.317021 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.317026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.318022 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.318024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.318946 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.318949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.319414 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.319415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.319564 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.320514 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.320525 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.322565 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.324880 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.324890 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.324894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.325341 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.325626 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.325627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.325628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.325888 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.326504 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.326510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.326675 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.326842 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.327177 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
I20260928 15:03:03.328620 32704 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39247
I20260928 15:03:03.328755 32704 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:03.328995 32704 heartbeater.cc:511] Master 127.25.123.62:39247 requested a full tablet report, sending...
I20260928 15:03:03.329911 32239 ts_manager.cc:194] Registered new tserver with Master: f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:03.330422 32239 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:48085
I20260928 15:03:03.341766 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.123.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.61.keytab.
Entry for principal oryx/127.25.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.61.keytab.
I20260928 15:03:03.397522 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:44691
--webserver_interface=127.25.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.61.keytab
--principal=oryx/127.25.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39247,127.25.123.61:44691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:03.540053 32709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:03.540649 32709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:03.540721 32709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:03.542902 32709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:03.542979 32709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:03.542997 32709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:03.543012 32709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:03.543026 32709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:03.547072 32709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39247,127.25.123.61:44691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.61.keytab
--principal=oryx/127.25.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:44691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
W20260928 15:03:03.547290 32709 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.123.62:39247,127.25.123.61:44691'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 15:03:03.547523 32709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:03.547806 32709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:03.555569 32715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.555624 32717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.555526 32714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:03.555507 32709 server_base.cc:1034] running on GCE node
I20260928 15:03:03.556206 32709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:03.556790 32709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:03.558018 32709 hybrid_clock.cc:648] HybridClock initialized: now 1790607783557979 us; error 64 us; skew 500 ppm
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:03.561599 32709 init.cc:377] Logged in from keytab as oryx/127.25.123.61@KRBTEST.COM (short username oryx)
I20260928 15:03:03.562184 32709 webserver.cc:460] Webserver started at http://127.25.123.61:41225/ using document root <none> and password file <none>
I20260928 15:03:03.562443 32709 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:03.562520 32709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:03.562685 32709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:03.563896 32709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/instance:
uuid: "c8a7934c5e654a0199dec86614b8daf5"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "eee03292bd590d5799230a1f16928911"
server_key_iv: "a06186e89bf7c1115cca0f1b89a31179"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.564302 32709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal/instance:
uuid: "c8a7934c5e654a0199dec86614b8daf5"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "eee03292bd590d5799230a1f16928911"
server_key_iv: "a06186e89bf7c1115cca0f1b89a31179"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.565910 32709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:03:03.567353 32724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.567768 32709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:03:03.567930 32709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
uuid: "c8a7934c5e654a0199dec86614b8daf5"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "eee03292bd590d5799230a1f16928911"
server_key_iv: "a06186e89bf7c1115cca0f1b89a31179"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.568035 32709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:03.600749 32709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:03.604833 32709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:03.605195 32709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:03.609859 32709 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.61:44691
I20260928 15:03:03.609958 308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.61:44691 every 8 connection(s)
I20260928 15:03:03.610466 32709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
I20260928 15:03:03.611253 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 32709
I20260928 15:03:03.612686 309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:03.616318 309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.123.61@KRBTEST.COM
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.61@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:03.630975 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.612980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.25.123.61:44691)
0928 15:03:03.613234 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.613244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.613792 (+ 548us) server_negotiation.cc:407] Connection header received
0928 15:03:03.614874 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.614889 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.614979 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.615174 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.616070 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.617560 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.618507 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.619513 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.619633 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.625212 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.625229 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.625230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.625263 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.627697 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.628381 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.628384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.628385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.628531 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.628941 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.628944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.628944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.629475 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.629624 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.630231 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.630665 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260928 15:03:03.631891 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.617175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:51936)
0928 15:03:03.617403 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.617448 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.617534 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.617752 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.617755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.618163 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.618524 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.618527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.622366 (+ 3839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.622368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.623553 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.623562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.624320 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.624322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.624481 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.625480 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.625485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.627697 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.630466 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.630474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.630476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.630754 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.630983 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.630984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.630984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.631083 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.631437 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.631444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.631505 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.631612 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.631746 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":152,"threads_started":1}
I20260928 15:03:03.632213 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.617249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.25.123.62:39247)
0928 15:03:03.617305 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.617307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.617533 (+ 226us) server_negotiation.cc:407] Connection header received
0928 15:03:03.617623 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.617625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.617637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.617707 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.619675 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.622182 (+ 2507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.623737 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.623996 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.624252 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.627896 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.627899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.627899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.627910 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.630322 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.630841 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.630842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.630843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.630912 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.631171 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.631173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.631173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.631359 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.631448 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.632051 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.632142 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:03:03.632988 309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.61@KRBTEST.COM for oryx/127.25.123.61@KRBTEST.COM
I20260928 15:03:03.644621 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.633657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.0.0.1:49456)
0928 15:03:03.633756 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.633771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.633839 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.634333 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.634334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.634718 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.634794 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.634799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.635921 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.635922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.636970 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.636974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.637615 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.637617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.637675 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.638748 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.638766 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.640757 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.643067 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.643068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.643069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.643346 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.643681 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.643682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.643682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.643847 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.644189 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.644191 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.644238 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.644361 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.644542 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 15:03:03.644972 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.633731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.25.123.61:44691)
0928 15:03:03.633775 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.633777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.633831 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:03:03.633994 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.633995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.634005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.634222 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.635169 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.635759 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.637145 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.637396 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.637493 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.640926 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.640928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.640929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.640937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.642931 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.643463 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.643464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.643465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.643571 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.643936 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.643937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.643938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.644103 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.644162 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.644492 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.644849 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:03:03.647063 309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Bootstrap starting.
I20260928 15:03:03.648010 309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:03.648307 309 log.cc:824] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:03.649873 309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: No bootstrap required, opened a new log
I20260928 15:03:03.651410 309 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } }
I20260928 15:03:03.651600 309 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:03.651649 309 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a7934c5e654a0199dec86614b8daf5, State: Initialized, Role: FOLLOWER
I20260928 15:03:03.651835 309 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } }
I20260928 15:03:03.652163 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } } }
I20260928 15:03:03.652241 309 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:03.652261 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:03:03.654843 326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:03:03.654928 326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:03:03.657763 32709 master_runner.cc:428] Detected that this master c8a7934c5e654a0199dec86614b8daf5 is joining an existing cluster
I20260928 15:03:03.657883 32709 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.123.61:44691
I20260928 15:03:03.668730 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.659168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.25.123.62:39247)
0928 15:03:03.659223 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.659226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.659379 (+ 153us) server_negotiation.cc:407] Connection header received
0928 15:03:03.659474 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.659475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.659488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.659540 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.660289 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.661403 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.662818 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.663130 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.663208 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.664914 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.664916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.664917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.664927 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.667160 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.667736 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.667738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.667738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.667810 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.668045 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.668047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.668047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.668222 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.668267 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.668520 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.668619 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:03:03.668766 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.659128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:51938)
0928 15:03:03.659287 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.659302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.659402 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.659556 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.659557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.660074 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.660189 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.660192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.661774 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.661777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.662695 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.662697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.663279 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.663280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.663421 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.664056 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.664062 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.664762 (+ 700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.667301 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.667302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.667303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.667624 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.667894 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.667894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.667895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.667957 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.668313 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.668314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.668360 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.668474 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.668615 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":55,"thread_start_us":97,"threads_started":1}
I20260928 15:03:03.669296 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.659424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.0.0.1:49468)
0928 15:03:03.659621 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.659632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.659678 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.659961 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.659962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.660169 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.660217 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.660219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.660987 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.660988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.662214 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.662218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.662927 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.662927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.663063 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.663858 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.663864 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.664762 (+ 898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.667806 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.667807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.667808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.668049 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.668530 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.668531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.668531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.668595 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.668966 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.668967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.669004 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.669072 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.669194 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260928 15:03:03.669402 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.659643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.25.123.61:44691)
0928 15:03:03.659769 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.659771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.659782 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:03.659871 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.659872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.659881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.659942 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.660329 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.660905 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.662381 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.662802 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.662912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.665032 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.665034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.665034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.665046 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.667721 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.668343 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.668344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.668344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.668416 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.668664 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.668665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.668665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.668870 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.668920 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.669108 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.669277 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":26}
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.62@KRBTEST.COM for oryx/127.25.123.61@KRBTEST.COM
I20260928 15:03:03.686291 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.673167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.25.123.61:44691)
0928 15:03:03.673232 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.673234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.673442 (+ 208us) server_negotiation.cc:407] Connection header received
0928 15:03:03.673750 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.673752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.673766 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.673885 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.674669 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.675321 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.676721 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.676978 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.677070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.680477 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.680479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.680480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.680504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.683521 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.684531 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.684532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.684533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.684610 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.685165 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.685166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.685167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.685413 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.685509 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.686031 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.686170 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 15:03:03.686671 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.673097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.0.0.1:49484)
0928 15:03:03.673331 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.673376 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.673471 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.674014 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.674020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.674490 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.674559 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.674562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.675770 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.675771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.676589 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.676592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.677068 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.677070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.677172 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.677905 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.677914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.680257 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.683972 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.683989 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.683992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.684363 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.684932 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.684933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.684934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.685029 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.685535 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.685618 (+ 83us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.685778 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.685951 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.686278 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260928 15:03:03.687675 32239 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.123.61:44691: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5"
member_type: NON_VOTER
last_known_addr {
host: "127.25.123.61"
port: 44691
}
attrs {
promote: true
}
}
dest_uuid: "33d29e49a3384726b7be308bc2c0553f"
cas_config_opid_index: -1
I20260928 15:03:03.689091 32239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } }
I20260928 15:03:03.689857 337 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } }
I20260928 15:03:03.690210 32751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:03.691638 32751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Refusing update from remote peer 33d29e49a3384726b7be308bc2c0553f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 15:03:03.691638 337 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.123.61:44691
I20260928 15:03:03.692433 337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } }
I20260928 15:03:03.692574 32709 master.cc:537] Master@127.25.123.61:44691 shutting down...
I20260928 15:03:03.692775 337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:03.694439 338 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:03:03.697566 32224 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 15:03:03.698274 32224 consensus_peers.cc:487] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f -> Peer c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61:44691): Couldn't send request to peer c8a7934c5e654a0199dec86614b8daf5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.25.123.60@KRBTEST.COM; defaulting to no policy
I20260928 15:03:03.698617 32709 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:03.698742 32709 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:03.698804 32709 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.60@KRBTEST.COM" created.
I20260928 15:03:03.703217 32709 master.cc:552] Master@127.25.123.61:44691 shutdown complete.
I20260928 15:03:03.703325 32709 master_runner.cc:305] Clearing existing system tablet
I20260928 15:03:03.703469 32709 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:03.703536 32709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:03.704257 32709 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260928 15:03:03.705094 343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.705236 32709 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:03.705289 32709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:03:03.705313 32709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
uuid: "c8a7934c5e654a0199dec86614b8daf5"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "eee03292bd590d5799230a1f16928911"
server_key_iv: "a06186e89bf7c1115cca0f1b89a31179"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.706774 32709 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:03.709343 32709 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:03:03.709524 32709 log.cc:1196] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 15:03:03.709870 32709 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Deleting consensus metadata
I20260928 15:03:03.710125 32709 master_runner.cc:315] Copying system tablet from 127.25.123.62:39247
I20260928 15:03:03.711184 32709 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.123.62:39247
I20260928 15:03:03.722409 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.711698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.25.123.62:39247)
0928 15:03:03.711756 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.711758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.711902 (+ 144us) server_negotiation.cc:407] Connection header received
0928 15:03:03.712039 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.712040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.712054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.712089 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.713104 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.714427 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.715494 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.715729 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.715820 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.717665 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:03.717667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:03.717667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:03.717676 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:03.720168 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.720886 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.720888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.720888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.720983 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:03.721390 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:03.721395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:03.721396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:03.721730 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:03.721880 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:03.722087 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 15:03:03.722258 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:03:03.722486 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.711589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:51942)
0928 15:03:03.711818 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.711830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.711949 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.712172 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.712172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.712562 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.712647 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.712657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.714591 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.714592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.715357 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.715360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:03.715818 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:03.715820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:03.715885 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:03.716993 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:03.716999 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:03.717350 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:03.720382 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.720384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.720385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.720774 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.721085 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:03.721086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:03.721086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:03.721150 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:03.721869 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:03.721871 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:03.721948 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 15:03:03.722039 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:03:03.722262 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
I20260928 15:03:03.722868 32260 tablet_copy_service.cc:140] P 33d29e49a3384726b7be308bc2c0553f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c8a7934c5e654a0199dec86614b8daf5 ({username='oryx', principal='oryx/127.25.123.61@KRBTEST.COM'} at 127.0.0.1:51942)
I20260928 15:03:03.723055 32260 tablet_copy_service.cc:161] P 33d29e49a3384726b7be308bc2c0553f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c8a7934c5e654a0199dec86614b8daf5 at {username='oryx', principal='oryx/127.25.123.61@KRBTEST.COM'} at 127.0.0.1:51942: session id = c8a7934c5e654a0199dec86614b8daf5-00000000000000000000000000000000
I20260928 15:03:03.724078 32260 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:03:03.725039 32709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:03:03.726581 32709 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: tablet copy: Starting download of 0 data blocks...
I20260928 15:03:03.726797 32709 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: tablet copy: Starting download of 1 WAL segments...
I20260928 15:03:03.729485 32709 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:03:03.730822 32260 tablet_copy_service.cc:342] P 33d29e49a3384726b7be308bc2c0553f: Request end of tablet copy session c8a7934c5e654a0199dec86614b8daf5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.123.61@KRBTEST.COM'} at 127.0.0.1:51942
I20260928 15:03:03.730890 32260 tablet_copy_service.cc:433] P 33d29e49a3384726b7be308bc2c0553f: ending tablet copy session c8a7934c5e654a0199dec86614b8daf5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c8a7934c5e654a0199dec86614b8daf5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.60.keytab.
Entry for principal oryx/127.25.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.60.keytab.
I20260928 15:03:03.734772 32709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:03:03.736546 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:45073
--webserver_interface=127.25.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.60.keytab
--principal=oryx/127.25.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:03.738029 350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.738300 351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.738525 353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:03.740710 32709 server_base.cc:1034] running on GCE node
I20260928 15:03:03.740876 32709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:03.741066 32709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:03.742337 32709 hybrid_clock.cc:648] HybridClock initialized: now 1790607783742317 us; error 32 us; skew 500 ppm
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:03.745254 32709 init.cc:377] Logged in from keytab as oryx/127.25.123.61@KRBTEST.COM (short username oryx)
I20260928 15:03:03.745644 32709 webserver.cc:460] Webserver started at http://127.25.123.61:41965/ using document root <none> and password file <none>
I20260928 15:03:03.745801 32709 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:03.745859 32709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:03.746820 32709 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:03.748768 362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.749017 32709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:03:03.749078 32709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
uuid: "c8a7934c5e654a0199dec86614b8daf5"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "eee03292bd590d5799230a1f16928911"
server_key_iv: "a06186e89bf7c1115cca0f1b89a31179"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.749173 32709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:03.761440 32709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:03.765146 32709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:03.769750 32709 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.61:44691
I20260928 15:03:03.770298 32709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
I20260928 15:03:03.773315 416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.61:44691 every 8 connection(s)
I20260928 15:03:03.774645 417 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:03:03.775609 417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Bootstrap starting.
I20260928 15:03:03.783138 417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:03:03.784549 417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: Bootstrap complete.
I20260928 15:03:03.784868 417 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } }
I20260928 15:03:03.784973 417 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8a7934c5e654a0199dec86614b8daf5, State: Initialized, Role: LEARNER
I20260928 15:03:03.785028 417 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } }
I20260928 15:03:03.785240 417 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:03.786914 431 catalog_manager.cc:1259] Loaded cluster ID: 51bc2c73f52c4394b43f49f3bf2a9094
I20260928 15:03:03.787011 431 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: loading cluster ID for follower catalog manager: success
I20260928 15:03:03.786616 420 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } }
I20260928 15:03:03.787611 420 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:03.788277 431 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: acquiring CA information for follower catalog manager: success
I20260928 15:03:03.788662 431 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:03:03.885145 32433 heartbeater.cc:503] Master 127.25.123.62:39247 was elected leader, sending a full tablet report...
W20260928 15:03:03.919076 356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:03.919446 356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:03.919493 356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:03.922374 356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:03.922513 356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:03.922539 356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:03.922566 356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:03.922600 356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:03.927430 356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.60.keytab
--principal=oryx/127.25.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:03.927855 356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:03.928189 356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:03.935427 436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.935529 439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:03.935624 437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:03.935803 356 server_base.cc:1034] running on GCE node
I20260928 15:03:03.936220 356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:03.937137 356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:03.938505 356 hybrid_clock.cc:648] HybridClock initialized: now 1790607783938487 us; error 108 us; skew 500 ppm
Sep 28 15:03:03 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:03.943471 356 init.cc:377] Logged in from keytab as oryx/127.25.123.60@KRBTEST.COM (short username oryx)
I20260928 15:03:03.944064 356 webserver.cc:460] Webserver started at http://127.25.123.60:45571/ using document root <none> and password file <none>
I20260928 15:03:03.944609 356 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:03.944753 356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:03.945232 356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:03.946594 356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/instance:
uuid: "df99afffce404a50a59d515f424b76b6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "fb3507bac1036042d9a056b03e7c77d8"
server_key_iv: "eded47ba21fb486a91d3006b46654ee1"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.947912 356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal/instance:
uuid: "df99afffce404a50a59d515f424b76b6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "fb3507bac1036042d9a056b03e7c77d8"
server_key_iv: "eded47ba21fb486a91d3006b46654ee1"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.950107 356 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260928 15:03:03.951140 446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:03.951558 356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:03.951828 356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
uuid: "df99afffce404a50a59d515f424b76b6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "fb3507bac1036042d9a056b03e7c77d8"
server_key_iv: "eded47ba21fb486a91d3006b46654ee1"
server_key_version: "encryptionkey@0"
I20260928 15:03:03.952032 356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:03.980103 356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:03.983196 356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:03.983403 356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:03.991438 356 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.60:45073
I20260928 15:03:03.991482 499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.60:45073 every 8 connection(s)
I20260928 15:03:03.992026 356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
I20260928 15:03:03.993657 500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:03.995584 500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:03:03.997236 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 356
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.60@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.123.60@KRBTEST.COM
I20260928 15:03:04.012290 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.996680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.25.123.62:39247)
0928 15:03:03.997008 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.997011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.997242 (+ 231us) server_negotiation.cc:407] Connection header received
0928 15:03:03.997887 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.997888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.997900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.997961 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.999137 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.000135 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.002395 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.002671 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.002795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.007341 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.007344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.007344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.007356 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.009648 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.010304 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.010306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.010306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.010378 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.010727 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.010729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.010729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.011048 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.011146 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.011622 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.011903 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 15:03:04.012569 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.996488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:51948)
0928 15:03:03.997101 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:03.997143 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:03.997766 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:03.998296 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:03.998300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:03.998579 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:03.998899 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:03.998909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.000341 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.000342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.002221 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.002226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.002840 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.002842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.002918 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.004759 (+ 1841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.004853 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.007011 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.009813 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.009820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.009823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.010148 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.010454 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.010455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.010455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.010623 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.011185 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.011191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.011411 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.011538 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.011746 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":297,"threads_started":1}
I20260928 15:03:04.013837 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:03.998063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.25.123.60:45073)
0928 15:03:03.998414 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:03.998420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:03.998463 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:03:03.998583 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:03.998586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:03.998621 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:03.998758 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:03.999524 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.001078 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.002409 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.002771 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.002906 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.008121 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.008132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.008133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.008142 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.010494 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.011195 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.011198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.011199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.011275 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.011571 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.011572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.011572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.011748 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.012424 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.013596 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.013718 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260928 15:03:04.013990 500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.60@KRBTEST.COM for oryx/127.25.123.61@KRBTEST.COM
I20260928 15:03:04.029426 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.014405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.0.0.1:49488)
0928 15:03:04.014819 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.014833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.014893 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.015488 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.015489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.015747 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.015947 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.015950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.017435 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.017437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.018501 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.018507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.019812 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.019813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.020111 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.021603 (+ 1492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.021610 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.023238 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.027555 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.027558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.027559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.027883 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.028303 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.028306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.028306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.028408 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.029056 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.029057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.029122 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.029209 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.029338 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391}
I20260928 15:03:04.030198 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.014524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.25.123.61:44691)
0928 15:03:04.015026 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.015029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.015039 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:04.015299 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.015300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.015321 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.015436 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.016268 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.017267 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.019104 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.019571 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.019810 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.023690 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.023694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.023694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.023705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.027280 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.028023 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.028024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.028025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.028109 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.028536 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.028539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.028539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.028781 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.029016 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.029455 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.029834 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":222,"threads_started":1}
I20260928 15:03:04.030766 500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607783, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.60@KRBTEST.COM for oryx/127.25.123.60@KRBTEST.COM
I20260928 15:03:04.043594 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.031247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.25.123.60:45073)
0928 15:03:04.031283 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.031285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.031293 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:04.031365 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.031366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.031375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.031416 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.031719 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.032272 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.033280 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.035250 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.035403 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.039271 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.039272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.039273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.039281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.041524 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.042124 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.042126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.042126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.042191 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.042472 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.042473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.042473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.042918 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.043088 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.043276 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.043440 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 15:03:04.043594 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.031138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.0.0.1:33162)
0928 15:03:04.031216 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.031229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.031281 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.031431 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.031432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.031591 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.031633 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.031635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.032420 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.032421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.033109 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.033112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.035402 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.035403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.035461 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.036512 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.036523 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.039128 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.041693 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.041694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.041695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.041999 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.042334 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.042335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.042335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.042402 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.043086 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.043088 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.043165 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.043247 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.043503 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":132}
I20260928 15:03:04.046715 500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Bootstrap starting.
I20260928 15:03:04.048003 500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:04.049283 500 log.cc:824] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:04.050428 500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: No bootstrap required, opened a new log
I20260928 15:03:04.052359 500 raft_consensus.cc:348] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } }
I20260928 15:03:04.053009 500 raft_consensus.cc:374] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:04.053082 500 raft_consensus.cc:729] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df99afffce404a50a59d515f424b76b6, State: Initialized, Role: FOLLOWER
I20260928 15:03:04.053510 500 consensus_queue.cc:260] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } }
I20260928 15:03:04.054365 507 sys_catalog.cc:455] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } } }
I20260928 15:03:04.054441 500 sys_catalog.cc:564] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:04.054491 507 sys_catalog.cc:458] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:03:04.057476 518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:03:04.058127 518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:03:04.064721 356 master_runner.cc:428] Detected that this master df99afffce404a50a59d515f424b76b6 is joining an existing cluster
I20260928 15:03:04.064879 356 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.123.60:45073
I20260928 15:03:04.067973 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 32302
I20260928 15:03:04.080582 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:33601
--local_ip_for_outbound_sockets=127.25.123.1
--tserver_master_addrs=127.25.123.62:39247
--webserver_port=42741
--webserver_interface=127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab
--principal=oryx/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:04.083405 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.067987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.25.123.60:45073)
0928 15:03:04.068027 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.068030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.069452 (+ 1422us) server_negotiation.cc:407] Connection header received
0928 15:03:04.069561 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.069562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.069572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.069600 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.070330 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.071097 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.072871 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.073177 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.073221 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.075453 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.075455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.075455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.075463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.080440 (+ 4977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.081048 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.081050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.081051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.081122 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.081353 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.081354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.081354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.081812 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.081887 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.083149 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.083262 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:03:04.083801 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.068059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.0.0.1:33166)
0928 15:03:04.069112 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.069127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.069487 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.069664 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.069664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.069970 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.070074 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.070077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.071210 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.071211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.072740 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.072742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.073725 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.073726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.073802 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.074950 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.074956 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.075344 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.080611 (+ 5267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.080613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.080614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.080945 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.081194 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.081194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.081195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.081279 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.081945 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.081946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.082159 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.083545 (+ 1386us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.083689 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":75,"thread_start_us":81,"threads_started":1}
I20260928 15:03:04.092809 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.069504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:51952)
0928 15:03:04.082011 (+ 12507us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.082031 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.082183 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.082451 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.082453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.082748 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.082865 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.082868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.084024 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.084025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.085003 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.085007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.085648 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.085648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.085745 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.086342 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.086348 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.087001 (+ 653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.090621 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.090622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.090623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.091014 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.091660 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.091662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.091663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.091808 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.092271 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.092272 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.092339 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.092470 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.092627 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12477,"thread_start_us":81,"threads_started":1}
I20260928 15:03:04.093472 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.076531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.25.123.62:39247)
0928 15:03:04.076711 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.076715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.082127 (+ 5412us) server_negotiation.cc:407] Connection header received
0928 15:03:04.082286 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.082288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.082304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.082340 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.082993 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.083878 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.085124 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.085524 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.085567 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.087135 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.087137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.087138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.087149 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.090467 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.091198 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.091200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.091201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.091465 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.091956 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.091957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.091958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.092110 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.092298 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.092850 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.093382 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 15:03:04.101859 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.067391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.25.123.61:44691)
0928 15:03:04.067452 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.067455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.067631 (+ 176us) server_negotiation.cc:407] Connection header received
0928 15:03:04.067830 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.067831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.067846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.067902 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.070203 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.071153 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.072866 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.073205 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.073243 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.097950 (+ 24707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.097953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.097953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.097962 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.100320 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.100744 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.100745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.100746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.100813 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.101035 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.101036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.101036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.101163 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.101259 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.101652 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.101752 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:03:04.101879 524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.067322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.0.0.1:49498)
0928 15:03:04.067542 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.067553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.067647 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.067962 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.067963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.069960 (+ 1997us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.070074 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.070077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.071246 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.071247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.072737 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.072741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.073746 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.073747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.073805 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.097130 (+ 23325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.097137 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.097736 (+ 599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.100438 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.100439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.100440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.100654 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.100893 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.100894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.100894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.100964 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.101489 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.101490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.101532 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.101611 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.101752 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":15,"thread_start_us":109,"threads_started":1}
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607782, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.62@KRBTEST.COM for oryx/127.25.123.60@KRBTEST.COM
I20260928 15:03:04.107733 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.097789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.0.0.1:33176)
0928 15:03:04.097849 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.097862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.097903 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.098107 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.098108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.098294 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.098362 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.098365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.099041 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.099043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.099855 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.099859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.100924 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.100925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.101026 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.101501 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.101508 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.103651 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.105989 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.105991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.105991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.106308 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.106944 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.106946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.106946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.107031 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.107349 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.107350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.107406 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.107494 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.107624 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:03:04.107729 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.097866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.25.123.60:45073)
0928 15:03:04.097906 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.097908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.097916 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:04.097976 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.097977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.097987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.098042 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.098459 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.098948 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.099963 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.100360 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.100847 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.103784 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.103786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.103786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.103794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.105874 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.106500 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.106503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.106504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.106785 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.107120 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.107121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.107122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.107268 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.107348 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.107533 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.107623 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:03:04.108942 32239 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.123.60:45073: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "df99afffce404a50a59d515f424b76b6"
member_type: NON_VOTER
last_known_addr {
host: "127.25.123.60"
port: 45073
}
attrs {
promote: true
}
}
dest_uuid: "33d29e49a3384726b7be308bc2c0553f"
cas_config_opid_index: 5
I20260928 15:03:04.109313 32239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } }
I20260928 15:03:04.109867 338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df99afffce404a50a59d515f424b76b6 (127.25.123.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.110988 338 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.123.60:45073
I20260928 15:03:04.111068 338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.111141 338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.112140 356 master.cc:537] Master@127.25.123.60:45073 shutting down...
I20260928 15:03:04.112236 32569 heartbeater.cc:503] Master 127.25.123.62:39247 was elected leader, sending a full tablet report...
I20260928 15:03:04.112583 473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:04.113543 473 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Refusing update from remote peer 33d29e49a3384726b7be308bc2c0553f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:03:04.114158 507 raft_consensus.cc:482] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:04.114265 507 raft_consensus.cc:504] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } }
I20260928 15:03:04.115497 507 leader_election.cc:288] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247), c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61:44691)
I20260928 15:03:04.116149 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df99afffce404a50a59d515f424b76b6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d29e49a3384726b7be308bc2c0553f" is_pre_election: true
I20260928 15:03:04.116350 390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df99afffce404a50a59d515f424b76b6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a7934c5e654a0199dec86614b8daf5" is_pre_election: true
I20260928 15:03:04.116739 390 raft_consensus.cc:1799] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 LEARNER]: Handling vote request from an unknown peer df99afffce404a50a59d515f424b76b6
I20260928 15:03:04.116832 390 raft_consensus.cc:2396] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate df99afffce404a50a59d515f424b76b6 for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260928 15:03:04.117254 448 leader_election.cc:302] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df99afffce404a50a59d515f424b76b6; no voters: 33d29e49a3384726b7be308bc2c0553f, c8a7934c5e654a0199dec86614b8daf5
I20260928 15:03:04.118433 337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:04.118835 507 raft_consensus.cc:2726] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 15:03:04.119403 32226 consensus_peers.cc:487] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f -> Peer df99afffce404a50a59d515f424b76b6 (127.25.123.60:45073): Couldn't send request to peer df99afffce404a50a59d515f424b76b6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:03:04.119895 356 raft_consensus.cc:2229] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:04.119993 356 raft_consensus.cc:2258] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:04.120043 356 tablet_replica.cc:330] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: stopping tablet replica
I20260928 15:03:04.124722 356 master.cc:552] Master@127.25.123.60:45073 shutdown complete.
I20260928 15:03:04.124794 356 master_runner.cc:305] Clearing existing system tablet
I20260928 15:03:04.124954 356 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:04.124994 356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:04.126947 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.111394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.0.0.1:49508)
0928 15:03:04.119267 (+ 7873us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.119282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.119341 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.119631 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.119632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.119634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:04.119687 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.119690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.121324 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.121327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.124180 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.124189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.126537 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.126539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.126651 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.126695 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.126785 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.126852 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7849}
I20260928 15:03:04.127321 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.111492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.25.123.61:44691)
0928 15:03:04.111567 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.111570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.119420 (+ 7850us) server_negotiation.cc:407] Connection header received
0928 15:03:04.119454 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.119455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.119481 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.119538 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:04.119780 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.121116 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.124331 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.125753 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.125909 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.126388 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.127172 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.127245 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:03:04.127614 390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 LEARNER]: Refusing update from remote peer 33d29e49a3384726b7be308bc2c0553f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:03:04.128089 337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 15:03:04.128845 390 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df99afffce404a50a59d515f424b76b6 (127.25.123.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.132372 420 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d29e49a3384726b7be308bc2c0553f. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.132535 420 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:04.132579 420 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: NON_VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: true } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.132625 420 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:04.132898 337 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: attempting to promote NON_VOTER c8a7934c5e654a0199dec86614b8daf5 to VOTER
I20260928 15:03:04.133150 337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } }
I20260928 15:03:04.134620 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.134480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.0.0.1:33186)
0928 15:03:04.134522 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.134578 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 15:03:04.134740 390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 LEARNER]: Refusing update from remote peer 33d29e49a3384726b7be308bc2c0553f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 15:03:04.134982 532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 15:03:04.135375 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.135249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.0.0.1:33188)
0928 15:03:04.135314 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.135343 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:03:04.135476 32226 consensus_peers.cc:487] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f -> Peer df99afffce404a50a59d515f424b76b6 (127.25.123.60:45073): Couldn't send request to peer df99afffce404a50a59d515f424b76b6. Status: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45073: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 15:03:04.135741 528 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.135905 390 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.137049 356 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.002s sys 0.000s
I20260928 15:03:04.137622 338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.137713 338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.137894 338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.137966 338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.138058 531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.138188 531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:04.138477 533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.138901 356 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:04.139008 356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:04.139051 356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
uuid: "df99afffce404a50a59d515f424b76b6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "fb3507bac1036042d9a056b03e7c77d8"
server_key_iv: "eded47ba21fb486a91d3006b46654ee1"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.142181 356 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:04.145481 356 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:03:04.145604 356 log.cc:1196] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 15:03:04.145846 356 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Deleting consensus metadata
I20260928 15:03:04.146016 356 master_runner.cc:315] Copying system tablet from 127.25.123.62:39247
I20260928 15:03:04.146853 356 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: tablet copy: Beginning tablet copy session from remote peer at address 127.25.123.62:39247
I20260928 15:03:04.156198 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.147423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.25.123.62:39247)
0928 15:03:04.147479 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.147481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.147636 (+ 155us) server_negotiation.cc:407] Connection header received
0928 15:03:04.147734 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.147735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.147746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.147832 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.148548 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.149212 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.150242 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.150530 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.150619 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.151993 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.151995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.151995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.152003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.154074 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.154734 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.154735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.154736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.154806 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.155106 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.155107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.155108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.155286 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.155373 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.155890 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.156048 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 15:03:04.156484 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.147382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:51960)
0928 15:03:04.147531 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.147542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.147656 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.147842 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.147843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.148080 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.148456 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.148459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.149309 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.149310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.149902 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.149904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.150626 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.150628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.150703 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.151490 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.151501 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.151854 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.154250 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.154252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.154252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.154623 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.154918 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.154919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.154919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.154991 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.155361 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.155363 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.155421 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.155813 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.156047 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 15:03:04.157694 32260 tablet_copy_service.cc:140] P 33d29e49a3384726b7be308bc2c0553f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer df99afffce404a50a59d515f424b76b6 ({username='oryx', principal='oryx/127.25.123.60@KRBTEST.COM'} at 127.0.0.1:51960)
I20260928 15:03:04.157823 32260 tablet_copy_service.cc:161] P 33d29e49a3384726b7be308bc2c0553f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer df99afffce404a50a59d515f424b76b6 at {username='oryx', principal='oryx/127.25.123.60@KRBTEST.COM'} at 127.0.0.1:51960: session id = df99afffce404a50a59d515f424b76b6-00000000000000000000000000000000
I20260928 15:03:04.158331 32260 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:03:04.159071 356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:03:04.161046 356 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: tablet copy: Starting download of 0 data blocks...
I20260928 15:03:04.161302 356 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: tablet copy: Starting download of 1 WAL segments...
I20260928 15:03:04.162845 356 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:03:04.164260 32260 tablet_copy_service.cc:342] P 33d29e49a3384726b7be308bc2c0553f: Request end of tablet copy session df99afffce404a50a59d515f424b76b6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.123.60@KRBTEST.COM'} at 127.0.0.1:51960
I20260928 15:03:04.164355 32260 tablet_copy_service.cc:433] P 33d29e49a3384726b7be308bc2c0553f: ending tablet copy session df99afffce404a50a59d515f424b76b6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer df99afffce404a50a59d515f424b76b6
I20260928 15:03:04.168838 356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:04.170981 540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.170981 543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.171356 356 server_base.cc:1034] running on GCE node
W20260928 15:03:04.171545 541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.171867 356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:04.172098 356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:04.173317 356 hybrid_clock.cc:648] HybridClock initialized: now 1790607784173309 us; error 34 us; skew 500 ppm
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:04.176134 356 init.cc:377] Logged in from keytab as oryx/127.25.123.60@KRBTEST.COM (short username oryx)
I20260928 15:03:04.176553 356 webserver.cc:460] Webserver started at http://127.25.123.60:38881/ using document root <none> and password file <none>
I20260928 15:03:04.176703 356 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:04.176743 356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:04.177448 356 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:04.178318 550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.178651 356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:03:04.178726 356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
uuid: "df99afffce404a50a59d515f424b76b6"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "fb3507bac1036042d9a056b03e7c77d8"
server_key_iv: "eded47ba21fb486a91d3006b46654ee1"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.178900 356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:04.198982 356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:04.201246 356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:04.205034 356 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.60:45073
I20260928 15:03:04.205451 602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.60:45073 every 8 connection(s)
I20260928 15:03:04.205513 356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
I20260928 15:03:04.206722 603 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:03:04.207572 603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Bootstrap starting.
I20260928 15:03:04.210532 603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:03:04.210846 603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Bootstrap complete.
I20260928 15:03:04.211097 603 raft_consensus.cc:348] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } }
I20260928 15:03:04.211203 603 raft_consensus.cc:729] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: df99afffce404a50a59d515f424b76b6, State: Initialized, Role: LEARNER
I20260928 15:03:04.211257 603 consensus_queue.cc:260] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } }
I20260928 15:03:04.211483 603 sys_catalog.cc:564] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:04.211473 606 sys_catalog.cc:455] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.211740 606 sys_catalog.cc:458] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:04.216055 617 catalog_manager.cc:1259] Loaded cluster ID: 51bc2c73f52c4394b43f49f3bf2a9094
I20260928 15:03:04.216337 617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: loading cluster ID for follower catalog manager: success
I20260928 15:03:04.217918 617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: acquiring CA information for follower catalog manager: success
I20260928 15:03:04.219005 617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:03:04.256755 527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:04.257140 527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:04.257187 527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:04.259658 527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:04.259945 527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:04.260087 527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:04.264667 527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab
--principal=oryx/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:33601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=42741
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:04.265105 527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:04.265455 527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:04.272616 625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.272635 623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.272701 622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.273478 527 server_base.cc:1034] running on GCE node
I20260928 15:03:04.273751 527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:04.274171 527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:04.275400 527 hybrid_clock.cc:648] HybridClock initialized: now 1790607784275337 us; error 85 us; skew 500 ppm
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:04.281858 527 init.cc:377] Logged in from keytab as oryx/127.25.123.1@KRBTEST.COM (short username oryx)
I20260928 15:03:04.282553 527 webserver.cc:460] Webserver started at http://127.25.123.1:42741/ using document root <none> and password file <none>
I20260928 15:03:04.282830 527 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:04.282933 527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:04.284595 527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:04.285532 632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.285841 527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:04.285944 527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "15bf9f5050714a10b3d6b720cc8ae6ad"
format_stamp: "Formatted at 2026-09-28 15:03:02 on dist-test-slave-qnz0"
server_key: "71f554bee5efacb4501bd3cbb90b769e"
server_key_iv: "065ee216446bc18febaa767c9d491d3b"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.286476 527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:04.325100 527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:04.328914 527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:04.329089 527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:04.329334 527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:04.329700 527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:04.329744 527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.329775 527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:04.329792 527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.332218 32704 heartbeater.cc:503] Master 127.25.123.62:39247 was elected leader, sending a full tablet report...
I20260928 15:03:04.338193 527 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:33601
I20260928 15:03:04.338546 748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:33601 every 8 connection(s)
I20260928 15:03:04.338801 527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:04.342761 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 527
I20260928 15:03:04.342931 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 32438
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.60@KRBTEST.COM
I20260928 15:03:04.350041 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:43599
--local_ip_for_outbound_sockets=127.25.123.2
--tserver_master_addrs=127.25.123.62:39247
--webserver_port=38795
--webserver_interface=127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab
--principal=oryx/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.61@KRBTEST.COM
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:04.355350 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.340282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:50985 (local address 127.25.123.62:39247)
0928 15:03:04.340382 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.340384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.340714 (+ 330us) server_negotiation.cc:407] Connection header received
0928 15:03:04.341534 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.341536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.341549 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.341583 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.343701 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.344599 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.346584 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.346893 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.346956 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.351421 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.351423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.351424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.351434 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.354048 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354501 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.354502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.354503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.354565 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354763 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.354764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.354765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.354898 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.354941 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.355133 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.355229 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:03:04.355351 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.340452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:57111 (local address 127.25.123.60:45073)
0928 15:03:04.340815 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.340816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.340827 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:04.341555 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.341556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.341570 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.341628 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.342678 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.345524 (+ 2846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.346504 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.346810 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.346879 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.351364 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.351365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.351366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.351374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.353953 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354426 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.354427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.354427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.354491 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354691 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.354692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.354693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.354846 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.354908 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.355147 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.355234 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":59,"threads_started":1}
I20260928 15:03:04.355598 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.339791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.25.123.1:57759)
0928 15:03:04.340621 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.340637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.341405 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.341768 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.341769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.342241 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.342563 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.342570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.346607 (+ 4037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.346608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.347609 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.347612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.348097 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.348099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.348185 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.348644 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.348649 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.350807 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.353534 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.353545 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.353548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.353969 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.354223 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.354224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.354391 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.354737 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.354744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.354943 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.355051 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.355275 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":36,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260928 15:03:04.355718 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.339791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.25.123.1:57111)
0928 15:03:04.340187 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.340282 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.341366 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.341758 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.341763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.342226 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.342563 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.342570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.345638 (+ 3068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.345639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.346394 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.346396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.346960 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.346961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.347130 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.348178 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.348182 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.351245 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.354055 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.354057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.354340 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.354560 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.354562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.354623 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.354936 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.354936 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.354984 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.355056 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.355566 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":350,"thread_start_us":99,"threads_started":1}
I20260928 15:03:04.356098 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.340259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:57759 (local address 127.25.123.61:44691)
0928 15:03:04.340339 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.340341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.340714 (+ 373us) server_negotiation.cc:407] Connection header received
0928 15:03:04.341531 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.341533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.341549 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.341583 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.342694 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.343495 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.347706 (+ 4211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.347984 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.348082 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.351009 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.351011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.351012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.351022 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.353416 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354068 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.354070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.354071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.354144 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354467 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.354468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.354469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.354636 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.354709 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.355886 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.355986 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:03:04.356190 749 heartbeater.cc:347] Connected to a master server at 127.25.123.60:45073
I20260928 15:03:04.356297 749 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.356305 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.340145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:50985)
0928 15:03:04.340634 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.340644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.341405 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.342229 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.342230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.342376 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.343589 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.343593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.344740 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.344741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.346397 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.346400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.347005 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.347005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.347129 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.348092 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.348103 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.351282 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.354132 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.354134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.354406 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.354639 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.354639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.354640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.354691 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.354986 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.354987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.355020 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.356027 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.356136 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260928 15:03:04.356550 749 heartbeater.cc:511] Master 127.25.123.60:45073 requested a full tablet report, sending...
I20260928 15:03:04.356948 750 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39247
I20260928 15:03:04.357002 750 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.357072 751 heartbeater.cc:347] Connected to a master server at 127.25.123.61:44691
I20260928 15:03:04.357076 750 heartbeater.cc:511] Master 127.25.123.62:39247 requested a full tablet report, sending...
I20260928 15:03:04.357138 751 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.357159 567 ts_manager.cc:194] Registered new tserver with Master: 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601)
I20260928 15:03:04.357205 751 heartbeater.cc:511] Master 127.25.123.61:44691 requested a full tablet report, sending...
I20260928 15:03:04.357393 32239 ts_manager.cc:194] Re-registered known tserver with Master: 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601)
I20260928 15:03:04.357539 379 ts_manager.cc:194] Registered new tserver with Master: 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601)
I20260928 15:03:04.357919 32239 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:50985
W20260928 15:03:04.454711 757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:04.455345 757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:04.455467 757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:04.458485 757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:04.458618 757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:04.458663 757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:04.462138 757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab
--principal=oryx/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:43599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=38795
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:04.462731 757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:04.463106 757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:04.471338 762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.471326 765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.471869 763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.472448 757 server_base.cc:1034] running on GCE node
I20260928 15:03:04.472761 757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:04.473105 757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:04.475003 757 hybrid_clock.cc:648] HybridClock initialized: now 1790607784474983 us; error 38 us; skew 500 ppm
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:04.480512 757 init.cc:377] Logged in from keytab as oryx/127.25.123.2@KRBTEST.COM (short username oryx)
I20260928 15:03:04.481586 757 webserver.cc:460] Webserver started at http://127.25.123.2:38795/ using document root <none> and password file <none>
I20260928 15:03:04.482177 757 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:04.482285 757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:04.484483 757 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:03:04.486100 773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.486472 757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:03:04.486594 757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "672eadd32ab8224fdae00557021c75bf"
server_key_iv: "47edebd1c2c490b032e2fe0487c2953e"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.487053 757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:04.502206 757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:04.505132 757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:04.505393 757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:04.505702 757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:04.506062 757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:04.506098 757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.506121 757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:04.506134 757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.513623 757 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:43599
I20260928 15:03:04.513767 888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:43599 every 8 connection(s)
I20260928 15:03:04.514150 757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:04.521015 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 757
I20260928 15:03:04.521174 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 32574
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.60@KRBTEST.COM
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.61@KRBTEST.COM
I20260928 15:03:04.528666 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.515005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:49855 (local address 127.25.123.60:45073)
0928 15:03:04.515059 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.515062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.515415 (+ 353us) server_negotiation.cc:407] Connection header received
0928 15:03:04.516030 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.516031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.516049 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.516108 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.518484 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.519489 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.520593 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.520885 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.520956 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.523926 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.523928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.523928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.523938 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.526771 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.527358 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.527360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.527360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.527421 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.527715 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.527716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.527716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.527876 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.527921 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.528351 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.528510 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:03:04.529340 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.515073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:41853 (local address 127.25.123.62:39247)
0928 15:03:04.515124 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.515126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.515357 (+ 231us) server_negotiation.cc:407] Connection header received
0928 15:03:04.516124 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.516126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.516139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.516233 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.517843 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.518447 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.519714 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.520004 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.520074 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.525858 (+ 5784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.525860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.525861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.525871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.527669 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.528107 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.528109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.528110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.528171 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.528432 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.528433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.528433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.528589 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.528637 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.528992 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.529182 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:03:04.529381 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.515024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.25.123.2:49855)
0928 15:03:04.515217 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.515264 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.515912 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.516252 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.516258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.517144 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.517540 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.517549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.519613 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.519614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.520489 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.520492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.521020 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.521021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.521092 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.521568 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.521573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.523752 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.526878 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.526891 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.526894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.527266 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.527490 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.527491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.527492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.527636 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.527970 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.527977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.528168 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.528846 (+ 678us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.529071 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":30,"thread_start_us":117,"threads_started":1}
I20260928 15:03:04.529836 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.515024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:41853)
0928 15:03:04.515218 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.515264 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.515971 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.516295 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.516296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.517139 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.517543 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.517644 (+ 101us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.518746 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.518748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.519616 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.519618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.520109 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.520110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.520183 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.521020 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.521025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.525710 (+ 4685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.527773 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.527774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.527775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.528013 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.528252 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.528252 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.528253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.528322 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.528690 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.528691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.528734 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.529491 (+ 757us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.529650 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":107,"thread_start_us":112,"threads_started":1}
I20260928 15:03:04.529974 889 heartbeater.cc:347] Connected to a master server at 127.25.123.60:45073
I20260928 15:03:04.530092 889 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.530308 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:39377
--local_ip_for_outbound_sockets=127.25.123.3
--tserver_master_addrs=127.25.123.62:39247
--webserver_port=43135
--webserver_interface=127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab
--principal=oryx/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:04.530349 889 heartbeater.cc:511] Master 127.25.123.60:45073 requested a full tablet report, sending...
I20260928 15:03:04.530745 890 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39247
I20260928 15:03:04.530831 890 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.530911 890 heartbeater.cc:511] Master 127.25.123.62:39247 requested a full tablet report, sending...
I20260928 15:03:04.531011 567 ts_manager.cc:194] Registered new tserver with Master: cfc7ef5285b74c54a9ac758bb7b0bebc (127.25.123.2:43599)
I20260928 15:03:04.531149 32239 ts_manager.cc:194] Re-registered known tserver with Master: cfc7ef5285b74c54a9ac758bb7b0bebc (127.25.123.2:43599)
I20260928 15:03:04.531677 32239 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:41853
I20260928 15:03:04.532857 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.515510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:58281 (local address 127.25.123.61:44691)
0928 15:03:04.515590 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.515592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.516192 (+ 600us) server_negotiation.cc:407] Connection header received
0928 15:03:04.516237 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.516238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.516251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.516312 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.517708 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.518354 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.519558 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.519871 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.519939 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.526925 (+ 6986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.526928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.526928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.526939 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.528722 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.529610 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.529613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.529613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.529706 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.532052 (+ 2346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.532054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.532054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.532361 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.532500 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.532687 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.532774 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:03:04.533056 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.515404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.25.123.2:58281)
0928 15:03:04.516116 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.516126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.516182 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.516348 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.516349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.517116 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.517540 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.517549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.518505 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.518508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.519442 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.519445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.520019 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.520020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.520172 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.520868 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.520881 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.526798 (+ 5917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.528946 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.528949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.528949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.529338 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.529821 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.529822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.529822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.531958 (+ 2136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.532489 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.532490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.532546 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.532647 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.532823 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":31,"thread_start_us":69,"threads_started":1}
I20260928 15:03:04.533526 891 heartbeater.cc:347] Connected to a master server at 127.25.123.61:44691
I20260928 15:03:04.533586 891 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.533679 891 heartbeater.cc:511] Master 127.25.123.61:44691 requested a full tablet report, sending...
I20260928 15:03:04.534135 379 ts_manager.cc:194] Registered new tserver with Master: cfc7ef5285b74c54a9ac758bb7b0bebc (127.25.123.2:43599)
W20260928 15:03:04.629197 896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:04.629449 896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:04.629477 896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:04.630808 896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:04.630869 896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:04.630901 896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:04.633121 896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab
--principal=oryx/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:39377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=43135
--tserver_master_addrs=127.25.123.62:39247,127.25.123.61:44691,127.25.123.60:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:04.633441 896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:04.633736 896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:04.639755 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:04.639771 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.639904 896 server_base.cc:1034] running on GCE node
W20260928 15:03:04.639771 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:04.640481 896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:04.640780 896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:04.642035 896 hybrid_clock.cc:648] HybridClock initialized: now 1790607784642069 us; error 95 us; skew 500 ppm
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:04.645220 896 init.cc:377] Logged in from keytab as oryx/127.25.123.3@KRBTEST.COM (short username oryx)
I20260928 15:03:04.645761 896 webserver.cc:460] Webserver started at http://127.25.123.3:43135/ using document root <none> and password file <none>
I20260928 15:03:04.646023 896 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:04.646116 896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:04.647897 896 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:03:04.648883 912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.649255 896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:04.649679 896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa"
format_stamp: "Formatted at 2026-09-28 15:03:03 on dist-test-slave-qnz0"
server_key: "6a83f9be73ec1cb3a4a8045277cf1859"
server_key_iv: "013b74cf116bfbd97f604d537a2bf040"
server_key_version: "encryptionkey@0"
I20260928 15:03:04.650146 896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:04.658286 896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:04.662079 896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:04.662642 896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:04.663039 896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:04.663609 896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:04.663681 896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.663731 896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:04.663759 896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:04.671402 896 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:39377
I20260928 15:03:04.671569 1026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:39377 every 8 connection(s)
I20260928 15:03:04.671892 896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:04.678337 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 896
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.61@KRBTEST.COM
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
Sep 28 15:03:04 dist-test-slave-qnz0 krb5kdc[32194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607784, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.60@KRBTEST.COM
I20260928 15:03:04.688977 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.673758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:39019 (local address 127.25.123.61:44691)
0928 15:03:04.673879 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.673882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.674709 (+ 827us) server_negotiation.cc:407] Connection header received
0928 15:03:04.676129 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.676130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.676146 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.676214 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.677274 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.677854 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.679637 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.679959 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680031 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.684151 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.684154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.684155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.684169 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.686903 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.687567 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.687569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.687569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.687656 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.688016 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.688017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.688018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.688166 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.688220 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.688754 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.688853 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:03:04.689484 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.673762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:60509)
0928 15:03:04.674541 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.674599 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.675984 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.676447 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.676448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.676798 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.677128 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.677133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.678966 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.678967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.679804 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.679806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.680672 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.680733 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.681752 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.681757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.684356 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.687567 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.687568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.687568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.687813 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.688051 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.688051 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.688052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.688106 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.688802 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.688804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.688850 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.688937 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.689193 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":4,"thread_start_us":266,"threads_started":1}
I20260928 15:03:04.690469 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.674024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:60509 (local address 127.25.123.62:39247)
0928 15:03:04.674081 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.674083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.674949 (+ 866us) server_negotiation.cc:407] Connection header received
0928 15:03:04.676284 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.676286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.676299 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.676436 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.677997 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.678837 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680135 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.680529 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680581 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.684548 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.684551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.684552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.684563 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.687451 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.687906 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.687907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.687907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.687974 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.688536 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.688537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.688538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.688704 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.688773 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.690222 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.690338 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 15:03:04.691036 1028 heartbeater.cc:347] Connected to a master server at 127.25.123.62:39247
I20260928 15:03:04.691184 1028 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.691458 1028 heartbeater.cc:511] Master 127.25.123.62:39247 requested a full tablet report, sending...
I20260928 15:03:04.691937 32239 ts_manager.cc:194] Re-registered known tserver with Master: f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:04.692551 32239 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:60509
I20260928 15:03:04.692904 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.673705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.25.123.3:39019)
0928 15:03:04.674549 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.674599 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.675984 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.676308 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.676312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.676798 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.677128 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.677133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.677979 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.677980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.679505 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.679507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.680560 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.680678 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.681612 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.681620 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.684027 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.686989 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.687008 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.687011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.687468 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.687741 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.687742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.687742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.687923 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.688276 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.688285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.688522 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.689080 (+ 558us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.691016 (+ 1936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":1803,"thread_start_us":339,"threads_started":1}
I20260928 15:03:04.693286 1029 heartbeater.cc:347] Connected to a master server at 127.25.123.61:44691
I20260928 15:03:04.693333 1029 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.693727 1029 heartbeater.cc:511] Master 127.25.123.61:44691 requested a full tablet report, sending...
I20260928 15:03:04.693982 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.679940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.25.123.61:44691)
0928 15:03:04.681054 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.681056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.681065 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:03:04.681087 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.681088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.681102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.681222 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.684021 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.685790 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.687086 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.687739 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.687819 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.689565 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.689567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.689567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.689575 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.692232 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.692746 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.692747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.692748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.692820 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.693091 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.693092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.693092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.693246 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.693357 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.693782 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.693881 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":101,"threads_started":1}
I20260928 15:03:04.694063 379 ts_manager.cc:194] Registered new tserver with Master: f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:04.695878 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.674299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:35539 (local address 127.25.123.60:45073)
0928 15:03:04.674445 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.674447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.675002 (+ 555us) server_negotiation.cc:407] Connection header received
0928 15:03:04.676171 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.676172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.676185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.676218 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.677432 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.678011 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680086 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.680430 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680509 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.691802 (+ 11293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.691804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.691805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.691813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.693869 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.694541 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.694544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.694545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.694681 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.694938 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.694940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.694940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.695296 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.695410 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.695624 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.695796 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 15:03:04.696514 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.674032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.25.123.3:35539)
0928 15:03:04.674858 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.674871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.676006 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.676314 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.676315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.677007 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:04.677127 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.677133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.678124 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.678125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.679503 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.679506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.680522 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.680523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.680678 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.681607 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:04.681620 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:04.691663 (+ 10043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:04.693968 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.693969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.693970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.694322 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.694775 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:04.694776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:04.694776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:04.694860 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:04.695423 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:04.695426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:04.695488 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.695581 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.696326 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":32,"thread_start_us":88,"threads_started":1}
I20260928 15:03:04.697065 1027 heartbeater.cc:347] Connected to a master server at 127.25.123.60:45073
I20260928 15:03:04.697145 1027 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:04.697211 1027 heartbeater.cc:511] Master 127.25.123.60:45073 requested a full tablet report, sending...
I20260928 15:03:04.697351 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.680489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.25.123.60:45073)
0928 15:03:04.681297 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.681300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.681308 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:04.681342 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.681343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.681359 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.681448 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.682715 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.683963 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.686037 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.686399 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.686483 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.690475 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.690478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.690478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.690489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.695681 (+ 5192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.696082 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.696084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.696084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.696168 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.696453 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.696454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.696455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.696631 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.696749 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.697205 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.697283 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":71,"threads_started":1}
I20260928 15:03:04.697510 567 ts_manager.cc:194] Registered new tserver with Master: f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:04.699726 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.680171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.25.123.62:39247)
0928 15:03:04.680304 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.680306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.687228 (+ 6922us) server_negotiation.cc:407] Connection header received
0928 15:03:04.687311 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.687312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.687330 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.687443 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.688204 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.689232 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.690505 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.690834 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.690914 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.695713 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.695715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.695716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.695725 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.697922 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.698377 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.698378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.698379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.698453 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.698722 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.698723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.698724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.698994 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.699094 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.699367 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.699532 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 15:03:04.702082 32239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:04.704999 32239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:04.714921 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.710629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.25.123.60:45073)
0928 15:03:04.710676 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.710678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.711005 (+ 327us) server_negotiation.cc:407] Connection header received
0928 15:03:04.711050 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.711051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.711076 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.711187 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:04.711337 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.712230 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.713926 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.714523 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.714631 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.714708 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.714794 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.714840 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:03:04.714926 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.710523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.0.0.1:33208)
0928 15:03:04.710691 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.710704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.710831 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.711196 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.711197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.711200 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:04.711260 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.711262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.712591 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.712592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.713832 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.713836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.714645 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.714647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.714723 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.714728 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.714787 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.714839 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":119,"threads_started":1}
I20260928 15:03:04.715523 577 raft_consensus.cc:1203] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 15:03:04.717777 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.711595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:39377 (local address 127.0.0.1:47090)
0928 15:03:04.711921 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.711930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.711969 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.713379 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.713381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.713386 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:04.713476 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.713478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.715372 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.715374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.716505 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.716509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.717480 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.717482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.717555 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.717560 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.717622 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.717669 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":115,"threads_started":1}
I20260928 15:03:04.718009 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.712715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.25.123.3:39377)
0928 15:03:04.713039 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.713044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.713057 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:04.713098 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.713100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.713119 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.713221 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:04.713877 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.714961 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.716618 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.717372 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.717447 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.717833 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.717865 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.717916 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":224,"threads_started":1}
I20260928 15:03:04.718480 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.711611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:33601 (local address 127.0.0.1:43718)
0928 15:03:04.712034 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.712043 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.712077 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.714218 (+ 2141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.714220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.714223 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:04.714279 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.714280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.715713 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.715715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.716951 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.716954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.718217 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.718219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.718295 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.718299 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.718364 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.718409 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":106,"threads_started":1}
I20260928 15:03:04.718889 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.711859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.25.123.1:33601)
0928 15:03:04.712221 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.712226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.712236 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:04.713992 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.713997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.714020 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.714114 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:04.714358 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.715602 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.717082 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.717796 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.718605 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.718700 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.718731 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.718785 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":108,"threads_started":1}
I20260928 15:03:04.718896 960 tablet_service.cc:1467] Processing CreateTablet for tablet edd558e772ff428ebd875a34384baabe (DEFAULT_TABLE table=test-table [id=8af198f1027949ac8d3a190292bf8e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:04.719105 1057 sys_catalog.cc:455] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d29e49a3384726b7be308bc2c0553f. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.719224 1057 sys_catalog.cc:458] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:03:04.719357 960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd558e772ff428ebd875a34384baabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:04.719624 671 tablet_service.cc:1467] Processing CreateTablet for tablet edd558e772ff428ebd875a34384baabe (DEFAULT_TABLE table=test-table [id=8af198f1027949ac8d3a190292bf8e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:04.719834 532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: NON_VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: true } } }
I20260928 15:03:04.719949 532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.719995 671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd558e772ff428ebd875a34384baabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:04.720749 1046 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: attempting to promote NON_VOTER df99afffce404a50a59d515f424b76b6 to VOTER
I20260928 15:03:04.720884 1046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:04.721506 390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Refusing update from remote peer 33d29e49a3384726b7be308bc2c0553f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260928 15:03:04.721642 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.711590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43599 (local address 127.0.0.1:36664)
0928 15:03:04.711781 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.711793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.711844 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.717328 (+ 5484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.717329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.717332 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:04.717401 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.717402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.719064 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.719065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.720284 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.720287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.721378 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.721379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.721461 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.721467 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.721505 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.721563 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260928 15:03:04.721741 532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 15:03:04.722208 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.711954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.25.123.2:43599)
0928 15:03:04.717134 (+ 5180us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.717139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.717149 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:04.717202 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.717204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.717220 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.717478 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:04.717544 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.718944 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.720451 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.721255 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.721740 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.721847 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.721880 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.721922 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5132,"thread_start_us":89,"threads_started":1}
I20260928 15:03:04.722958 577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 LEARNER]: Refusing update from remote peer 33d29e49a3384726b7be308bc2c0553f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260928 15:03:04.723769 1046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, df99afffce404a50a59d515f424b76b6 (127.25.123.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.724092 822 tablet_service.cc:1467] Processing CreateTablet for tablet edd558e772ff428ebd875a34384baabe (DEFAULT_TABLE table=test-table [id=8af198f1027949ac8d3a190292bf8e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:04.724695 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd558e772ff428ebd875a34384baabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:04.724639 390 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, df99afffce404a50a59d515f424b76b6 (127.25.123.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.725926 1064 tablet_bootstrap.cc:492] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Bootstrap starting.
I20260928 15:03:04.727021 1064 tablet_bootstrap.cc:654] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:04.727430 1064 log.cc:824] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:04.727376 532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 15:03:04.727835 1066 tablet_bootstrap.cc:492] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Bootstrap starting.
I20260928 15:03:04.728094 577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, df99afffce404a50a59d515f424b76b6 (127.25.123.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.728590 1066 tablet_bootstrap.cc:654] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:04.728600 1046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.728682 1046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.728756 1046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.728838 1046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.729003 1066 log.cc:824] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:04.729518 1057 sys_catalog.cc:455] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.729632 1057 sys_catalog.cc:458] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:04.730039 1065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.730167 1065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:04.730178 1066 tablet_bootstrap.cc:492] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: No bootstrap required, opened a new log
I20260928 15:03:04.730150 1064 tablet_bootstrap.cc:492] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: No bootstrap required, opened a new log
I20260928 15:03:04.730260 1064 ts_tablet_manager.cc:1397] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 15:03:04.730511 1066 ts_tablet_manager.cc:1397] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:04.730809 1063 tablet_bootstrap.cc:492] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Bootstrap starting.
I20260928 15:03:04.731624 1063 tablet_bootstrap.cc:654] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:04.732115 1063 log.cc:824] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:04.732780 1064 raft_consensus.cc:348] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.732995 1064 raft_consensus.cc:374] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:04.733042 1064 raft_consensus.cc:729] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bf9f5050714a10b3d6b720cc8ae6ad, State: Initialized, Role: FOLLOWER
I20260928 15:03:04.733125 1066 raft_consensus.cc:348] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.733513 1063 tablet_bootstrap.cc:492] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: No bootstrap required, opened a new log
I20260928 15:03:04.733520 1066 raft_consensus.cc:374] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:04.733559 1066 raft_consensus.cc:729] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc7ef5285b74c54a9ac758bb7b0bebc, State: Initialized, Role: FOLLOWER
I20260928 15:03:04.733589 1063 ts_tablet_manager.cc:1397] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:04.733453 1064 consensus_queue.cc:260] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.733712 1066 consensus_queue.cc:260] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.734040 1066 ts_tablet_manager.cc:1428] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:04.734274 890 heartbeater.cc:503] Master 127.25.123.62:39247 was elected leader, sending a full tablet report...
I20260928 15:03:04.734202 1064 ts_tablet_manager.cc:1428] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 15:03:04.735025 750 heartbeater.cc:503] Master 127.25.123.62:39247 was elected leader, sending a full tablet report...
I20260928 15:03:04.736726 1063 raft_consensus.cc:348] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.737016 1063 raft_consensus.cc:374] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:04.737069 1063 raft_consensus.cc:729] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1858f2db1c34dcd99f6f0cf6f2771aa, State: Initialized, Role: FOLLOWER
I20260928 15:03:04.737202 1063 consensus_queue.cc:260] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.737668 1063 ts_tablet_manager.cc:1428] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:03:04.737668 1028 heartbeater.cc:503] Master 127.25.123.62:39247 was elected leader, sending a full tablet report...
I20260928 15:03:04.739112 1070 raft_consensus.cc:482] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:04.739223 1070 raft_consensus.cc:504] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.739809 1070 leader_election.cc:288] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601), f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:04.743185 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.740170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:47341 (local address 127.25.123.1:33601)
0928 15:03:04.740459 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.740461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.740468 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:03:04.740493 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.740494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.740506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.740563 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:04.740742 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.741248 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.742226 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.742802 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.742878 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.742888 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.743070 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.743108 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260928 15:03:04.743196 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.739964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:33601 (local address 127.25.123.2:47341)
0928 15:03:04.740006 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.740018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.740064 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.740582 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.740583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.740585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:04.740658 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.740662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.741341 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.741342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.742103 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.742105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.742881 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.742882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.742992 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.742995 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.743065 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.743114 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 15:03:04.743736 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.740169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:39377 (local address 127.25.123.2:52439)
0928 15:03:04.740311 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:04.740326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:04.740352 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:04.740830 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:04.740831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:04.740834 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:04.740895 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.740897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.741781 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.741782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.742800 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:04.742804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.743411 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.743412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:04.743520 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.743524 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:04.743594 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 15:03:04.743650 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 15:03:04.743649 702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd558e772ff428ebd875a34384baabe" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" is_pre_election: true
I20260928 15:03:04.743762 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.740262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:52439 (local address 127.25.123.3:39377)
0928 15:03:04.740336 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.740338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.740360 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:03:04.740652 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.740653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.740664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.740809 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:04.740998 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.741490 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.742881 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.743256 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.743366 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.743376 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.743672 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.743714 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:03:04.743861 702 raft_consensus.cc:2454] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 0.
I20260928 15:03:04.744086 774 leader_election.cc:302] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bf9f5050714a10b3d6b720cc8ae6ad, cfc7ef5285b74c54a9ac758bb7b0bebc; no voters:
I20260928 15:03:04.744269 1070 raft_consensus.cc:2781] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:04.744333 1070 raft_consensus.cc:482] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:04.744261 980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd558e772ff428ebd875a34384baabe" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" is_pre_election: true
I20260928 15:03:04.744365 1070 raft_consensus.cc:3037] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:04.744447 980 raft_consensus.cc:2454] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 0.
I20260928 15:03:04.745285 1070 raft_consensus.cc:504] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.745467 1070 leader_election.cc:288] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 election: Requested vote from peers 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601), f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:04.745654 702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd558e772ff428ebd875a34384baabe" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad"
I20260928 15:03:04.745754 702 raft_consensus.cc:3037] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:04.745782 980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd558e772ff428ebd875a34384baabe" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa"
I20260928 15:03:04.745877 980 raft_consensus.cc:3037] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:04.746563 702 raft_consensus.cc:2454] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 1.
I20260928 15:03:04.746801 980 raft_consensus.cc:2454] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 1.
I20260928 15:03:04.746922 774 leader_election.cc:302] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bf9f5050714a10b3d6b720cc8ae6ad, cfc7ef5285b74c54a9ac758bb7b0bebc; no voters:
I20260928 15:03:04.747030 1070 raft_consensus.cc:2781] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:04.747215 1070 raft_consensus.cc:686] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 LEADER]: Becoming Leader. State: Replica: cfc7ef5285b74c54a9ac758bb7b0bebc, State: Running, Role: LEADER
I20260928 15:03:04.747447 1070 consensus_queue.cc:237] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:04.748615 32239 catalog_manager.cc:5495] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc7ef5285b74c54a9ac758bb7b0bebc (127.25.123.2). New cstate: current_term: 1 leader_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:04.749341 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33d29e49a3384726b7be308bc2c0553f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:04.749454 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:04.765378 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.761827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.25.123.2:43599)
0928 15:03:04.762064 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.762066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.762076 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:04.762118 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.762119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.762141 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.762267 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:04.762426 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.762945 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.764453 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.764668 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.764755 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.764973 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:04.765109 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:04.765214 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.765238 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.765270 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
W20260928 15:03:04.766073 892 tablet.cc:2369] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:04.768806 702 raft_consensus.cc:1261] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Refusing update from remote peer cfc7ef5285b74c54a9ac758bb7b0bebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:04.768805 980 raft_consensus.cc:1261] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Refusing update from remote peer cfc7ef5285b74c54a9ac758bb7b0bebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:04.769275 1073 consensus_queue.cc:1035] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:04.769487 1070 consensus_queue.cc:1035] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:04.774575 1078 mvcc.cc:205] Tried to move back new op lower bound from 7334329486406488064 to 7334329486326566912. Current Snapshot: MvccSnapshot[applied={T|T < 7334329486406488064}]
I20260928 15:03:04.774714 1076 mvcc.cc:205] Tried to move back new op lower bound from 7334329486406488064 to 7334329486326566912. Current Snapshot: MvccSnapshot[applied={T|T < 7334329486406488064}]
I20260928 15:03:04.776971 1077 mvcc.cc:205] Tried to move back new op lower bound from 7334329486406488064 to 7334329486326566912. Current Snapshot: MvccSnapshot[applied={T|T < 7334329486406488064}]
I20260928 15:03:04.779244 32239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:04.779356 32239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:04.781517 32239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33d29e49a3384726b7be308bc2c0553f: Sending DeleteTablet for 3 replicas of tablet edd558e772ff428ebd875a34384baabe
I20260928 15:03:04.782114 671 tablet_service.cc:1514] Processing DeleteTablet for tablet edd558e772ff428ebd875a34384baabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:04 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:43718
I20260928 15:03:04.783057 1085 tablet_replica.cc:330] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: stopping tablet replica
I20260928 15:03:04.783404 1085 raft_consensus.cc:2229] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:04.783699 960 tablet_service.cc:1514] Processing DeleteTablet for tablet edd558e772ff428ebd875a34384baabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:04 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47090
I20260928 15:03:04.783759 1085 raft_consensus.cc:2258] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:04.784267 822 tablet_service.cc:1514] Processing DeleteTablet for tablet edd558e772ff428ebd875a34384baabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:04 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36664
I20260928 15:03:04.784628 1088 tablet_replica.cc:330] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: stopping tablet replica
I20260928 15:03:04.784761 1088 raft_consensus.cc:2229] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:04.784865 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 32208
I20260928 15:03:04.784894 1088 raft_consensus.cc:2258] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:04.784895 1085 ts_tablet_manager.cc:1905] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:04.786369 1087 tablet_replica.cc:330] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: stopping tablet replica
I20260928 15:03:04.786580 1087 raft_consensus.cc:2229] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:04.787273 1087 raft_consensus.cc:2258] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:04.787981 1087 ts_tablet_manager.cc:1905] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:04.788650 1085 ts_tablet_manager.cc:1918] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:04.789404 1085 log.cc:1196] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/edd558e772ff428ebd875a34384baabe
I20260928 15:03:04.790014 1085 ts_tablet_manager.cc:1939] T edd558e772ff428ebd875a34384baabe P 15bf9f5050714a10b3d6b720cc8ae6ad: Deleting consensus metadata
I20260928 15:03:04.791962 1087 ts_tablet_manager.cc:1918] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:04.792204 1087 log.cc:1196] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/edd558e772ff428ebd875a34384baabe
I20260928 15:03:04.792608 1087 ts_tablet_manager.cc:1939] T edd558e772ff428ebd875a34384baabe P f1858f2db1c34dcd99f6f0cf6f2771aa: Deleting consensus metadata
W20260928 15:03:04.794775 637 connection.cc:535] server connection from 127.0.0.1:43718 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 15:03:04.794845 916 connection.cc:535] server connection from 127.0.0.1:47090 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:03:04.796550 1088 ts_tablet_manager.cc:1905] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:04.798926 1088 ts_tablet_manager.cc:1918] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:04.799026 1088 log.cc:1196] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/edd558e772ff428ebd875a34384baabe
I20260928 15:03:04.799341 1088 ts_tablet_manager.cc:1939] T edd558e772ff428ebd875a34384baabe P cfc7ef5285b74c54a9ac758bb7b0bebc: Deleting consensus metadata
W20260928 15:03:04.800014 776 connection.cc:441] server connection from 127.0.0.1:36664 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36664 (request call id 1) could send its response
I20260928 15:03:04.806792 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.784210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.25.123.61:44691)
0928 15:03:04.784267 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.784269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.796612 (+ 12343us) server_negotiation.cc:407] Connection header received
0928 15:03:04.796698 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.796699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.796733 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.796829 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.797645 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.798654 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.799840 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.800137 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.800247 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.802196 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.802198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.802198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.802207 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.804234 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.805256 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.805257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.805257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.805328 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.805547 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.805548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.805548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.805860 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.805973 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.806573 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.806664 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 15:03:04.808063 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.797189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.25.123.60:45073)
0928 15:03:04.797246 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.797248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.797671 (+ 423us) server_negotiation.cc:407] Connection header received
0928 15:03:04.797766 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.797768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.797778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.797809 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.799197 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.800011 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.801330 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.801973 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.802086 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.804098 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.804100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.804101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.804111 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.806422 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.806849 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.806850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.806851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.806909 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.807432 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.807433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.807433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.807578 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.807665 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.807877 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.807963 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:03:04.809427 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.800501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.25.123.61:44691)
0928 15:03:04.800546 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.800549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.800960 (+ 411us) server_negotiation.cc:407] Connection header received
0928 15:03:04.801070 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.801071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.801083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.801114 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.801777 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.802607 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.803757 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.804082 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.804132 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.805884 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.805886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.805886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.805893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.807852 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.808425 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.808426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.808427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.808495 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.808760 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.808761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.808761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.808914 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.808995 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.809228 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.809329 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:03:04.816120 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:04.784277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.25.123.60:45073)
0928 15:03:04.784316 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:04.784318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:04.807452 (+ 23134us) server_negotiation.cc:407] Connection header received
0928 15:03:04.807575 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:04.807576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:04.807591 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:04.807628 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:04.808484 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.809274 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.810352 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:04.810561 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:04.810674 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:04.812253 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:04.812255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:04.812256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:04.812265 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:04.814510 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.815219 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.815221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.815221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.815289 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:04.815489 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:04.815490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:04.815490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:04.815620 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:04.815717 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:04.815897 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 15:03:04.815979 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:03:05.775529 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:05.774974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:45103)
0928 15:03:05.775260 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:05.775463 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":119,"threads_started":1}
W20260928 15:03:05.775949 750 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:39247 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:05.778584 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:05.778154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:51287)
0928 15:03:05.778438 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:05.778520 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":76,"threads_started":1}
I20260928 15:03:05.778828 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:05.778081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:60483)
0928 15:03:05.778573 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:05.778695 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":70,"threads_started":1}
W20260928 15:03:05.778908 890 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:39247 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
W20260928 15:03:05.779096 1028 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:39247 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:06.282728 1099 raft_consensus.cc:482] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 33d29e49a3384726b7be308bc2c0553f)
I20260928 15:03:06.283004 1099 raft_consensus.cc:504] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:06.283512 1099 leader_election.cc:288] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247), c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61:44691)
I20260928 15:03:06.283878 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.283565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52456)
0928 15:03:06.283722 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.283809 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260928 15:03:06.284020 1102 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 33d29e49a3384726b7be308bc2c0553f)
W20260928 15:03:06.284219 552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:06.284325 1102 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:06.284732 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.284625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52472)
0928 15:03:06.284675 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.284705 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 15:03:06.284793 1102 leader_election.cc:288] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247), df99afffce404a50a59d515f424b76b6 (127.25.123.60:45073)
W20260928 15:03:06.284837 552 leader_election.cc:334] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247): Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:06.285143 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.284880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52480)
0928 15:03:06.285028 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.285090 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
W20260928 15:03:06.285308 364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:06.285924 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.285799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52496)
0928 15:03:06.285831 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.285857 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 15:03:06.286010 364 leader_election.cc:334] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247): Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:06.289400 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.284949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45073 (local address 127.0.0.1:33216)
0928 15:03:06.285154 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.285165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:06.285214 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:06.285469 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:06.285470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:06.285472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:06.285545 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:06.285547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.286481 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.286482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:06.287847 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:06.287850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.288996 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.288997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:06.289094 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:06.289100 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:03:06.289178 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:03:06.289279 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260928 15:03:06.289431 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.285193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.25.123.60:45073)
0928 15:03:06.285334 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:06.285336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:06.285344 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:06.285365 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:06.285366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:06.285378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:06.285451 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:06.285618 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.286375 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.288037 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.288795 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.288979 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:06.288995 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:06.289214 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 15:03:06.289333 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260928 15:03:06.289505 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.283987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.25.123.61:44691)
0928 15:03:06.284124 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:06.284126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:06.284142 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:03:06.284284 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:06.284287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:06.284311 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:06.284524 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:06.285118 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.286565 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.288324 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.288997 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.289067 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:06.289076 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:06.289278 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 15:03:06.289331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260928 15:03:06.289438 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.283801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:44691 (local address 127.0.0.1:49550)
0928 15:03:06.283957 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.283974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:06.284064 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:06.284539 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:06.284540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:06.284543 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:06.284731 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:06.284735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.286707 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.286708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:06.288106 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:06.288112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:06.289105 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:06.289106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:06.289187 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:06.289191 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:06.289251 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 15:03:06.289340 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":29,"thread_start_us":93,"threads_started":1}
I20260928 15:03:06.289728 577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8a7934c5e654a0199dec86614b8daf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df99afffce404a50a59d515f424b76b6" is_pre_election: true
I20260928 15:03:06.289791 390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df99afffce404a50a59d515f424b76b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8a7934c5e654a0199dec86614b8daf5" is_pre_election: true
I20260928 15:03:06.289896 390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df99afffce404a50a59d515f424b76b6 in term 1.
I20260928 15:03:06.289918 577 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a7934c5e654a0199dec86614b8daf5 in term 1.
I20260928 15:03:06.290072 551 leader_election.cc:302] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8a7934c5e654a0199dec86614b8daf5, df99afffce404a50a59d515f424b76b6; no voters: 33d29e49a3384726b7be308bc2c0553f
I20260928 15:03:06.290210 1099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:03:06.290258 1099 raft_consensus.cc:482] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 33d29e49a3384726b7be308bc2c0553f)
I20260928 15:03:06.290241 365 leader_election.cc:302] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8a7934c5e654a0199dec86614b8daf5, df99afffce404a50a59d515f424b76b6; no voters: 33d29e49a3384726b7be308bc2c0553f
I20260928 15:03:06.290292 1099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:03:06.290364 1102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:03:06.290407 1102 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 33d29e49a3384726b7be308bc2c0553f)
I20260928 15:03:06.290427 1102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:03:06.291301 1099 raft_consensus.cc:504] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:06.291368 1102 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:06.291492 1099 leader_election.cc:288] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 election: Requested vote from peers 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247), c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61:44691)
I20260928 15:03:06.291674 1102 leader_election.cc:288] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [CANDIDATE]: Term 2 election: Requested vote from peers 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247), df99afffce404a50a59d515f424b76b6 (127.25.123.60:45073)
I20260928 15:03:06.291716 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.291582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52500)
0928 15:03:06.291644 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.291683 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:03:06.291741 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.291622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52512)
0928 15:03:06.291664 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.291700 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 15:03:06.291776 390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df99afffce404a50a59d515f424b76b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8a7934c5e654a0199dec86614b8daf5"
I20260928 15:03:06.291857 390 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df99afffce404a50a59d515f424b76b6 in current term 2: Already voted for candidate c8a7934c5e654a0199dec86614b8daf5 in this term.
I20260928 15:03:06.291867 577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8a7934c5e654a0199dec86614b8daf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df99afffce404a50a59d515f424b76b6"
I20260928 15:03:06.291946 577 raft_consensus.cc:2379] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a7934c5e654a0199dec86614b8daf5 in current term 2: Already voted for candidate df99afffce404a50a59d515f424b76b6 in this term.
I20260928 15:03:06.292091 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.291975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52514)
0928 15:03:06.292034 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.292067 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 15:03:06.292165 364 leader_election.cc:334] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247): Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:06.292222 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.292097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:52524)
0928 15:03:06.292165 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.292193 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 15:03:06.292282 365 leader_election.cc:302] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8a7934c5e654a0199dec86614b8daf5; no voters: 33d29e49a3384726b7be308bc2c0553f, df99afffce404a50a59d515f424b76b6
W20260928 15:03:06.292294 552 leader_election.cc:334] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247): Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:06.292433 551 leader_election.cc:302] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df99afffce404a50a59d515f424b76b6; no voters: 33d29e49a3384726b7be308bc2c0553f, c8a7934c5e654a0199dec86614b8daf5
I20260928 15:03:06.292536 1099 raft_consensus.cc:2726] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 15:03:06.292562 1102 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 15:03:06.777184 1109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.776610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:51775)
0928 15:03:06.777025 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.777103 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":301,"threads_started":1}
I20260928 15:03:06.779630 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.779341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:59555)
0928 15:03:06.779524 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.779569 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260928 15:03:06.779683 1111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:06.779379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:51149)
0928 15:03:06.779525 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:06.779569 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260928 15:03:07.778872 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.778527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:58601)
0928 15:03:07.778725 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.778778 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":116,"threads_started":1}
I20260928 15:03:07.780453 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.780162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:49003)
0928 15:03:07.780306 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.780353 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260928 15:03:07.780532 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.780156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:49135)
0928 15:03:07.780436 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.780477 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":175,"threads_started":1}
I20260928 15:03:07.880525 1115 raft_consensus.cc:482] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:07.880663 1115 raft_consensus.cc:504] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:07.880947 1115 leader_election.cc:288] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247), c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61:44691)
I20260928 15:03:07.881292 390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df99afffce404a50a59d515f424b76b6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8a7934c5e654a0199dec86614b8daf5" is_pre_election: true
I20260928 15:03:07.881385 390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df99afffce404a50a59d515f424b76b6 in term 2.
I20260928 15:03:07.881546 551 leader_election.cc:302] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8a7934c5e654a0199dec86614b8daf5, df99afffce404a50a59d515f424b76b6; no voters:
I20260928 15:03:07.881667 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.881111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:58024)
0928 15:03:07.881223 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.881281 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260928 15:03:07.881731 1115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 15:03:07.881829 1115 raft_consensus.cc:482] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:07.881858 1115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:03:07.882531 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.882350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:58038)
0928 15:03:07.882450 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.882486 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 15:03:07.882635 552 leader_election.cc:334] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247): Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:07.882753 1115 raft_consensus.cc:504] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:07.882933 1115 leader_election.cc:288] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 3 election: Requested vote from peers 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247), c8a7934c5e654a0199dec86614b8daf5 (127.25.123.61:44691)
I20260928 15:03:07.883109 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.883024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:58050)
0928 15:03:07.883059 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.883085 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 15:03:07.883131 390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df99afffce404a50a59d515f424b76b6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8a7934c5e654a0199dec86614b8daf5"
I20260928 15:03:07.883240 390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:03:07.883576 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.883459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:58058)
0928 15:03:07.883519 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.883550 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 15:03:07.883675 552 leader_election.cc:334] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247): Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
I20260928 15:03:07.884032 390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df99afffce404a50a59d515f424b76b6 in term 3.
I20260928 15:03:07.884217 551 leader_election.cc:302] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8a7934c5e654a0199dec86614b8daf5, df99afffce404a50a59d515f424b76b6; no voters: 33d29e49a3384726b7be308bc2c0553f
I20260928 15:03:07.884377 1115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:03:07.884516 1115 raft_consensus.cc:686] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [term 3 LEADER]: Becoming Leader. State: Replica: df99afffce404a50a59d515f424b76b6, State: Running, Role: LEADER
I20260928 15:03:07.884613 1115 consensus_queue.cc:237] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } }
I20260928 15:03:07.885008 1118 sys_catalog.cc:455] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df99afffce404a50a59d515f424b76b6. Latest consensus state: current_term: 3 leader_uuid: "df99afffce404a50a59d515f424b76b6" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:07.885092 1118 sys_catalog.cc:458] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:07.885340 1120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:07.885998 1120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:07.886103 1120 catalog_manager.cc:1259] Loaded cluster ID: 51bc2c73f52c4394b43f49f3bf2a9094
I20260928 15:03:07.886159 1120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:07.886446 1120 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:07.886559 1120 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Loaded TSK: 0
I20260928 15:03:07.886711 1120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:07.924336 565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:07.926635 565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:07.928049 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.927887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:58086)
0928 15:03:07.927940 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.928009 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 15:03:07.928068 390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [term 3 FOLLOWER]: Refusing update from remote peer df99afffce404a50a59d515f424b76b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 15:03:07.928364 1115 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 15:03:07.928678 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.928457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.0.0.1:58098)
0928 15:03:07.928551 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.928623 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:03:07.928858 552 consensus_peers.cc:487] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 -> Peer 33d29e49a3384726b7be308bc2c0553f (127.25.123.62:39247): Couldn't send request to peer 33d29e49a3384726b7be308bc2c0553f. Status: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 15:03:07.929724 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df99afffce404a50a59d515f424b76b6. Latest consensus state: current_term: 3 leader_uuid: "df99afffce404a50a59d515f424b76b6" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:07.930104 1121 mvcc.cc:205] Tried to move back new op lower bound from 7334329499350245376 to 7334329499176087552. Current Snapshot: MvccSnapshot[applied={T|T < 7334329499350245376}]
I20260928 15:03:07.930145 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:07.930207 1118 sys_catalog.cc:455] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "df99afffce404a50a59d515f424b76b6" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:07.930322 1118 sys_catalog.cc:458] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:07.930701 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "df99afffce404a50a59d515f424b76b6" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "33d29e49a3384726b7be308bc2c0553f" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39247 } } peers { permanent_uuid: "c8a7934c5e654a0199dec86614b8daf5" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "df99afffce404a50a59d515f424b76b6" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45073 } attrs { promote: false } } }
I20260928 15:03:07.930855 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a7934c5e654a0199dec86614b8daf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:03:07.937764 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.933153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43599 (local address 127.0.0.1:58374)
0928 15:03:07.933472 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.933484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:07.933538 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:07.933958 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:07.933959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:07.933961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:07.934012 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:07.934014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.934964 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.934966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:07.936392 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:07.936396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.937437 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.937438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:07.937511 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:07.937516 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:07.937608 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:03:07.937669 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":39,"thread_start_us":239,"threads_started":1}
I20260928 15:03:07.937835 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.933502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.25.123.2:43599)
0928 15:03:07.933766 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:07.933768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:07.933775 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:03:07.933800 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:07.933801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:07.933818 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:07.933903 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:07.934086 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.934864 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.936538 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.937282 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.937436 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:07.937456 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:07.937618 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 15:03:07.937674 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":65,"threads_started":1}
I20260928 15:03:07.938261 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.933357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.25.123.3:39377)
0928 15:03:07.933733 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:07.933735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:07.933870 (+ 135us) server_negotiation.cc:407] Connection header received
0928 15:03:07.933905 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:07.933906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:07.933917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:07.934009 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:07.934163 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.934971 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.936705 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.937521 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.937622 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:07.937649 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:07.937904 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 15:03:07.937968 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":75,"threads_started":1}
I20260928 15:03:07.937994 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.933085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:33601 (local address 127.0.0.1:32790)
0928 15:03:07.933141 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.933155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:07.933207 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:07.933639 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:07.933640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:07.933641 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:07.933708 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:07.933710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.934602 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.934603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:07.936221 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:07.936226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.937666 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.937668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:07.937749 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:07.937754 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:07.937789 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 15:03:07.937856 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":62}
I20260928 15:03:07.938608 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.933085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:39377 (local address 127.0.0.1:46886)
0928 15:03:07.933790 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.933801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:07.933858 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:07.934027 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:07.934028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:07.934030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:07.934085 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:07.934086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.935135 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.935137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:07.936530 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:07.936533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.937685 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.937686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:07.937753 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:07.937756 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:03:07.937805 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 15:03:07.937856 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":507,"thread_start_us":73,"threads_started":1}
I20260928 15:03:07.938602 822 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7df6e00c474efc84d3039b0bf0f5cc (DEFAULT_TABLE table=test-table [id=25e7615de51247c696371f4f4b4afb10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:07.938781 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7df6e00c474efc84d3039b0bf0f5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:07.938973 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.933221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.25.123.1:33601)
0928 15:03:07.933355 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:07.933358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:07.933381 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:03:07.933413 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:07.933414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:07.933428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:07.933634 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:07.933815 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.934503 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.936453 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:07.937308 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:07.938781 (+ 1473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:07.938807 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:07.938835 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 15:03:07.938906 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260928 15:03:07.939034 960 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7df6e00c474efc84d3039b0bf0f5cc (DEFAULT_TABLE table=test-table [id=25e7615de51247c696371f4f4b4afb10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:07.939282 960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7df6e00c474efc84d3039b0bf0f5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:07.940558 671 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7df6e00c474efc84d3039b0bf0f5cc (DEFAULT_TABLE table=test-table [id=25e7615de51247c696371f4f4b4afb10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:07.940835 671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7df6e00c474efc84d3039b0bf0f5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:07.941084 1133 tablet_bootstrap.cc:492] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: Bootstrap starting.
I20260928 15:03:07.941530 1134 tablet_bootstrap.cc:492] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Bootstrap starting.
I20260928 15:03:07.941584 1133 tablet_bootstrap.cc:654] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:07.942211 1134 tablet_bootstrap.cc:654] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:07.942431 1133 tablet_bootstrap.cc:492] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: No bootstrap required, opened a new log
I20260928 15:03:07.942513 1133 ts_tablet_manager.cc:1397] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:07.942700 1133 raft_consensus.cc:348] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:07.942829 1133 raft_consensus.cc:374] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:07.942857 1133 raft_consensus.cc:729] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc7ef5285b74c54a9ac758bb7b0bebc, State: Initialized, Role: FOLLOWER
I20260928 15:03:07.942911 1133 consensus_queue.cc:260] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:07.943089 1133 ts_tablet_manager.cc:1428] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:07.943217 1134 tablet_bootstrap.cc:492] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: No bootstrap required, opened a new log
I20260928 15:03:07.943488 1134 ts_tablet_manager.cc:1397] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:07.943722 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.943527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:49447)
0928 15:03:07.943653 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.943691 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 15:03:07.943708 1134 raft_consensus.cc:348] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:07.943795 1134 raft_consensus.cc:374] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:07.943815 1134 raft_consensus.cc:729] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1858f2db1c34dcd99f6f0cf6f2771aa, State: Initialized, Role: FOLLOWER
I20260928 15:03:07.943966 1134 consensus_queue.cc:260] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:07.944113 889 heartbeater.cc:503] Master 127.25.123.60:45073 was elected leader, sending a full tablet report...
I20260928 15:03:07.944233 1134 ts_tablet_manager.cc:1428] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:07.944326 1137 tablet_bootstrap.cc:492] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad: Bootstrap starting.
I20260928 15:03:07.944561 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.944459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:45971)
0928 15:03:07.944500 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.944528 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 15:03:07.944847 1137 tablet_bootstrap.cc:654] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:07.944900 1027 heartbeater.cc:503] Master 127.25.123.60:45073 was elected leader, sending a full tablet report...
I20260928 15:03:07.945169 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.945072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:45995)
0928 15:03:07.945109 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.945140 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 15:03:07.945693 1137 tablet_bootstrap.cc:492] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad: No bootstrap required, opened a new log
I20260928 15:03:07.945745 1137 ts_tablet_manager.cc:1397] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 15:03:07.945868 1137 raft_consensus.cc:348] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:07.945916 1137 raft_consensus.cc:374] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:07.945930 1137 raft_consensus.cc:729] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bf9f5050714a10b3d6b720cc8ae6ad, State: Initialized, Role: FOLLOWER
I20260928 15:03:07.945999 1137 consensus_queue.cc:260] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:07.946168 1137 ts_tablet_manager.cc:1428] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:07.946561 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:07.946459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:37645)
0928 15:03:07.946507 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:07.946534 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 15:03:08.094264 752 tablet.cc:2369] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:03:08.176604 1030 tablet.cc:2369] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:08.207546 1138 raft_consensus.cc:482] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:08.207682 1138 raft_consensus.cc:504] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:08.207919 1138 leader_election.cc:288] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601), f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:08.208287 980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7df6e00c474efc84d3039b0bf0f5cc" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" is_pre_election: true
I20260928 15:03:08.208287 702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7df6e00c474efc84d3039b0bf0f5cc" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" is_pre_election: true
I20260928 15:03:08.208427 702 raft_consensus.cc:2454] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 0.
I20260928 15:03:08.208442 980 raft_consensus.cc:2454] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 0.
I20260928 15:03:08.208595 775 leader_election.cc:302] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc7ef5285b74c54a9ac758bb7b0bebc, f1858f2db1c34dcd99f6f0cf6f2771aa; no voters:
I20260928 15:03:08.208856 1138 raft_consensus.cc:2781] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:08.208946 1138 raft_consensus.cc:482] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:08.208981 1138 raft_consensus.cc:3037] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:08.209885 1138 raft_consensus.cc:504] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:08.210098 1138 leader_election.cc:288] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 election: Requested vote from peers 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601), f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377)
I20260928 15:03:08.210332 980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7df6e00c474efc84d3039b0bf0f5cc" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa"
I20260928 15:03:08.210387 702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7df6e00c474efc84d3039b0bf0f5cc" candidate_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad"
I20260928 15:03:08.210449 980 raft_consensus.cc:3037] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:08.210484 702 raft_consensus.cc:3037] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:08.211241 702 raft_consensus.cc:2454] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 1.
I20260928 15:03:08.211267 980 raft_consensus.cc:2454] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc7ef5285b74c54a9ac758bb7b0bebc in term 1.
I20260928 15:03:08.211457 774 leader_election.cc:302] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bf9f5050714a10b3d6b720cc8ae6ad, cfc7ef5285b74c54a9ac758bb7b0bebc; no voters:
I20260928 15:03:08.211572 1138 raft_consensus.cc:2781] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:08.211786 1138 raft_consensus.cc:686] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 LEADER]: Becoming Leader. State: Replica: cfc7ef5285b74c54a9ac758bb7b0bebc, State: Running, Role: LEADER
I20260928 15:03:08.211851 1138 consensus_queue.cc:237] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } }
I20260928 15:03:08.212656 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.212359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:40593)
0928 15:03:08.212449 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.212495 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 15:03:08.212626 565 catalog_manager.cc:5495] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc7ef5285b74c54a9ac758bb7b0bebc (127.25.123.2). New cstate: current_term: 1 leader_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc7ef5285b74c54a9ac758bb7b0bebc" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:08.231881 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.229013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.25.123.2:43599)
0928 15:03:08.229078 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:08.229080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:08.229089 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:03:08.229119 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:08.229120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:08.229152 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:08.229209 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:08.229628 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:08.230133 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:08.231036 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:08.231287 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:08.231365 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:08.231584 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:08.231627 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:08.231676 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:08.231787 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 15:03:08.231811 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:03:08.233078 980 raft_consensus.cc:1261] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Refusing update from remote peer cfc7ef5285b74c54a9ac758bb7b0bebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:08.233327 702 raft_consensus.cc:1261] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Refusing update from remote peer cfc7ef5285b74c54a9ac758bb7b0bebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:08.233453 1138 consensus_queue.cc:1035] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1858f2db1c34dcd99f6f0cf6f2771aa" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:08.233771 1143 consensus_queue.cc:1035] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bf9f5050714a10b3d6b720cc8ae6ad" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:08.234560 749 heartbeater.cc:503] Master 127.25.123.60:45073 was elected leader, sending a full tablet report...
I20260928 15:03:08.234926 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.234772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:38969)
0928 15:03:08.234836 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.234882 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 15:03:08.235890 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.235742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:60733)
0928 15:03:08.235803 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.235848 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:03:08.237287 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.236992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:53955)
0928 15:03:08.237175 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.237236 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260928 15:03:08.237317 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.236707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.3:58387)
0928 15:03:08.237238 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.237276 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510}
I20260928 15:03:08.238515 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.238174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.2:33955)
0928 15:03:08.238411 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.238464 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260928 15:03:08.239006 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.238879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39247 (local address 127.25.123.1:41429)
0928 15:03:08.238924 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.238961 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 15:03:08.241475 565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:08.241624 565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:08.243652 565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df99afffce404a50a59d515f424b76b6: Sending DeleteTablet for 3 replicas of tablet 3a7df6e00c474efc84d3039b0bf0f5cc
I20260928 15:03:08.244112 960 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7df6e00c474efc84d3039b0bf0f5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.25.123.60@KRBTEST.COM'} at 127.0.0.1:46886
I20260928 15:03:08.244169 671 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7df6e00c474efc84d3039b0bf0f5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.25.123.60@KRBTEST.COM'} at 127.0.0.1:32790
I20260928 15:03:08.244436 1155 tablet_replica.cc:330] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: stopping tablet replica
I20260928 15:03:08.244509 1155 raft_consensus.cc:2229] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:08.244589 822 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7df6e00c474efc84d3039b0bf0f5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.25.123.60@KRBTEST.COM'} at 127.0.0.1:58374
I20260928 15:03:08.244632 1156 tablet_replica.cc:330] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad: stopping tablet replica
I20260928 15:03:08.244699 1156 raft_consensus.cc:2229] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:08.244755 1156 raft_consensus.cc:2258] T 3a7df6e00c474efc84d3039b0bf0f5cc P 15bf9f5050714a10b3d6b720cc8ae6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:08.244905 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 527
I20260928 15:03:08.245982 1157 tablet_replica.cc:330] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: stopping tablet replica
I20260928 15:03:08.246035 1157 raft_consensus.cc:2229] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:08.246089 1157 raft_consensus.cc:2258] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:08.246417 1157 ts_tablet_manager.cc:1905] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:08.248483 1155 raft_consensus.cc:2258] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:08.248624 1157 ts_tablet_manager.cc:1918] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:08.248781 1157 log.cc:1196] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/3a7df6e00c474efc84d3039b0bf0f5cc
I20260928 15:03:08.249083 1157 ts_tablet_manager.cc:1939] T 3a7df6e00c474efc84d3039b0bf0f5cc P cfc7ef5285b74c54a9ac758bb7b0bebc: Deleting consensus metadata
I20260928 15:03:08.249763 553 catalog_manager.cc:4841] TS cfc7ef5285b74c54a9ac758bb7b0bebc (127.25.123.2:43599): tablet 3a7df6e00c474efc84d3039b0bf0f5cc (table test-table [id=25e7615de51247c696371f4f4b4afb10]) successfully deleted
I20260928 15:03:08.252825 1155 ts_tablet_manager.cc:1905] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:08.253072 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 757
I20260928 15:03:08.253232 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:08.253045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:33601 (local address 127.0.0.1:32800)
0928 15:03:08.253097 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:08.253186 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:33601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 15:03:08.255988 551 catalog_manager.cc:4584] TS 15bf9f5050714a10b3d6b720cc8ae6ad (127.25.123.1:33601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a7df6e00c474efc84d3039b0bf0f5cc: Network error: Client connection negotiation failed: client connection to 127.25.123.1:33601: connect: Connection refused (error 111)
I20260928 15:03:08.256454 1155 ts_tablet_manager.cc:1918] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:08.256529 1155 log.cc:1196] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/3a7df6e00c474efc84d3039b0bf0f5cc
I20260928 15:03:08.256772 1155 ts_tablet_manager.cc:1939] T 3a7df6e00c474efc84d3039b0bf0f5cc P f1858f2db1c34dcd99f6f0cf6f2771aa: Deleting consensus metadata
I20260928 15:03:08.257665 552 catalog_manager.cc:4841] TS f1858f2db1c34dcd99f6f0cf6f2771aa (127.25.123.3:39377): tablet 3a7df6e00c474efc84d3039b0bf0f5cc (table test-table [id=25e7615de51247c696371f4f4b4afb10]) successfully deleted
I20260928 15:03:08.262096 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 896
I20260928 15:03:08.268704 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 32709
I20260928 15:03:08.275256 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 356
2026-09-28T15:03:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7376 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:08 dist-test-slave-qnz0 krb5kdc[1160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:08 dist-test-slave-qnz0 krb5kdc[1160](info): set up 2 sockets
Sep 28 15:03:08 dist-test-slave-qnz0 krb5kdc[1160](info): commencing operation
krb5kdc: starting...
W20260928 15:03:09.444178 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:03:09 dist-test-slave-qnz0 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab.
Entry for principal oryx/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab.
I20260928 15:03:09.602319 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45455
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab
--principal=oryx/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:45455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:09.700672 1173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:09.700970 1173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:09.701015 1173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:09.702829 1173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:09.702934 1173 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:03:09.702965 1173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:09.702988 1173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:09.703016 1173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:09.703044 1173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:09.705180 1173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:45455
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.62.keytab
--principal=oryx/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:09.705549 1173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:09.705824 1173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:03:09.712656 1173 server_base.cc:1034] running on GCE node
W20260928 15:03:09.712762 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:09.712638 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:09.712896 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:09.713784 1173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:09.714246 1173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:09.715524 1173 hybrid_clock.cc:648] HybridClock initialized: now 1790607789715422 us; error 137 us; skew 500 ppm
Sep 28 15:03:09 dist-test-slave-qnz0 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:09.719908 1173 init.cc:377] Logged in from keytab as oryx/127.25.123.62@KRBTEST.COM (short username oryx)
I20260928 15:03:09.720516 1173 webserver.cc:460] Webserver started at http://127.25.123.62:40163/ using document root <none> and password file <none>
I20260928 15:03:09.720772 1173 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:09.720845 1173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:09.721014 1173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:09.723173 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "501fe93f19c240bab873ad0bb0f640c1"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-qnz0"
server_key: "9b9db784dc19afbf0a4f619572155cd9"
server_key_iv: "67a831b8b43067a96ef52228f1f35cba"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.723681 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "501fe93f19c240bab873ad0bb0f640c1"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-qnz0"
server_key: "9b9db784dc19afbf0a4f619572155cd9"
server_key_iv: "67a831b8b43067a96ef52228f1f35cba"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.725576 1173 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260928 15:03:09.727468 1188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.727929 1173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:03:09.728076 1173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "501fe93f19c240bab873ad0bb0f640c1"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-qnz0"
server_key: "9b9db784dc19afbf0a4f619572155cd9"
server_key_iv: "67a831b8b43067a96ef52228f1f35cba"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.728183 1173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:09.751442 1173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:09.755600 1173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:09.755960 1173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:09.763612 1173 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:45455
I20260928 15:03:09.763821 1251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:45455 every 8 connection(s)
I20260928 15:03:09.764113 1173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:09.765102 1252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:09.767728 1252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Bootstrap starting.
I20260928 15:03:09.768486 1252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:09.768827 1252 log.cc:824] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:09.769726 1252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: No bootstrap required, opened a new log
I20260928 15:03:09.770285 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 1173
I20260928 15:03:09.770589 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:03:09.770875 26092 external_mini_cluster.cc:1428] Setting key b1b79daef633859520654bbf583f76f3
I20260928 15:03:09.773443 1252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501fe93f19c240bab873ad0bb0f640c1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45455 } }
I20260928 15:03:09.773677 1252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:09.773721 1252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501fe93f19c240bab873ad0bb0f640c1, State: Initialized, Role: FOLLOWER
I20260928 15:03:09.773958 1252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501fe93f19c240bab873ad0bb0f640c1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45455 } }
I20260928 15:03:09.774091 1252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:09.774132 1252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:09.774184 1252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:09.775094 1252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501fe93f19c240bab873ad0bb0f640c1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45455 } }
I20260928 15:03:09.775202 1252 leader_election.cc:302] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 501fe93f19c240bab873ad0bb0f640c1; no voters:
I20260928 15:03:09.775391 1252 leader_election.cc:288] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:09.775475 1257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:09.775719 1252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:09.775743 1257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [term 1 LEADER]: Becoming Leader. State: Replica: 501fe93f19c240bab873ad0bb0f640c1, State: Running, Role: LEADER
I20260928 15:03:09.775980 1257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501fe93f19c240bab873ad0bb0f640c1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45455 } }
I20260928 15:03:09.777089 1259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "501fe93f19c240bab873ad0bb0f640c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501fe93f19c240bab873ad0bb0f640c1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45455 } } }
I20260928 15:03:09.777223 1259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:09.777767 1258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 501fe93f19c240bab873ad0bb0f640c1. Latest consensus state: current_term: 1 leader_uuid: "501fe93f19c240bab873ad0bb0f640c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501fe93f19c240bab873ad0bb0f640c1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45455 } } }
I20260928 15:03:09.778468 1258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1 [sys.catalog]: This master's current role is: LEADER
W20260928 15:03:09.778183 1271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:03:09.778566 1271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:03:09.778699 1272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:09.779101 1272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:03:09 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:09.785673 1272 catalog_manager.cc:1347] Generated new cluster ID: 8c882ebe08ad4eb6b25d78bdc4871268
I20260928 15:03:09.785758 1272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:09.791702 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:09.771507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.25.123.62:45455)
0928 15:03:09.771715 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:09.771724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:09.771756 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:03:09.773475 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:09.773495 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:09.773619 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:09.773934 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:09.774927 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.777030 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.779063 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:09.779838 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:09.779985 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:09.784519 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:09.784536 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:09.784537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:09.784560 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:09.788728 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:09.789272 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:09.789276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:09.789277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:09.789425 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:09.789832 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:09.789834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:09.789834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:09.790231 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:09.790389 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:09.790825 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 15:03:09.791413 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
I20260928 15:03:09.795620 1272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:09.796520 1272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:09.807889 1272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Generated new TSK 0
I20260928 15:03:09.808174 1272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab.
Entry for principal oryx/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab.
I20260928 15:03:09.845764 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45455
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab
--principal=oryx/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:09.953186 1278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:09.953625 1278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:09.953671 1278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:09.956784 1278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:09.956889 1278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:09.956925 1278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:03:09.956948 1278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:09.959249 1278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab
--principal=oryx/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:09.959599 1278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:09.959913 1278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:09.966279 1284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:09.966388 1285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:09.966467 1278 server_base.cc:1034] running on GCE node
W20260928 15:03:09.966619 1287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:09.966893 1278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:09.967185 1278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:09.968477 1278 hybrid_clock.cc:648] HybridClock initialized: now 1790607789968153 us; error 332 us; skew 500 ppm
Sep 28 15:03:09 dist-test-slave-qnz0 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:09.971899 1278 init.cc:377] Logged in from keytab as oryx/127.25.123.1@KRBTEST.COM (short username oryx)
I20260928 15:03:09.972728 1278 webserver.cc:460] Webserver started at http://127.25.123.1:34997/ using document root <none> and password file <none>
I20260928 15:03:09.973016 1278 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:09.973093 1278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:09.973271 1278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:09.974480 1278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "c6d66b08cc2f45e58aee35fddaef3811"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-qnz0"
server_key: "60422a57503e9f792597d44dcd9b8dcd"
server_key_iv: "ad4726b32bb14362a76e39983b721163"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.974898 1278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "c6d66b08cc2f45e58aee35fddaef3811"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-qnz0"
server_key: "60422a57503e9f792597d44dcd9b8dcd"
server_key_iv: "ad4726b32bb14362a76e39983b721163"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.977232 1278 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 15:03:09.979847 1294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:09.980340 1278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:09.980688 1278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "c6d66b08cc2f45e58aee35fddaef3811"
format_stamp: "Formatted at 2026-09-28 15:03:09 on dist-test-slave-qnz0"
server_key: "60422a57503e9f792597d44dcd9b8dcd"
server_key_iv: "ad4726b32bb14362a76e39983b721163"
server_key_version: "encryptionkey@0"
I20260928 15:03:09.980835 1278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:10.007083 1278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:10.010056 1278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:10.010538 1278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:10.012209 1278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:10.012341 1278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.012513 1278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:10.012617 1278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:10.027190 1278 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:40075
I20260928 15:03:10.027235 1410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:40075 every 8 connection(s)
I20260928 15:03:10.027767 1278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:10.030730 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.012622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:44587 (local address 127.25.123.62:45455)
0928 15:03:10.012692 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.012695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.012941 (+ 246us) server_negotiation.cc:407] Connection header received
0928 15:03:10.013496 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.013497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.013518 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.013610 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.014819 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.017392 (+ 2573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.020422 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.020724 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.020852 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.025933 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.025936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.025936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.025957 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.029034 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.029587 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.029589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.029589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.029661 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.029950 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.029951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.029951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.030091 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.030169 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.030505 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.030617 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:03:10.030959 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.012600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45455 (local address 127.25.123.1:44587)
0928 15:03:10.012797 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.012842 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.013378 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.013785 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.013789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.014091 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.014407 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.014414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.017708 (+ 3294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.017711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.020021 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.020031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.020896 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.020898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.021086 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.021979 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.022004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.025708 (+ 3704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.029191 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.029201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.029204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.029489 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.029745 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.029746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.029746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.029871 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.030172 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.030180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.030350 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.030466 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.030659 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260928 15:03:10.031579 1411 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45455
I20260928 15:03:10.031693 1411 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:10.031921 1411 heartbeater.cc:511] Master 127.25.123.62:45455 requested a full tablet report, sending...
I20260928 15:03:10.033092 1205 ts_manager.cc:194] Registered new tserver with Master: c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075)
I20260928 15:03:10.034085 1205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:44587
I20260928 15:03:10.034312 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 1278
I20260928 15:03:10.034456 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:10.034746 26092 external_mini_cluster.cc:1428] Setting key 4a68007d7a14b5530fbdfe67e7b1a7e7
I20260928 15:03:10.042652 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.033133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:52193 (local address 127.25.123.62:45455)
0928 15:03:10.033196 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.033199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.033278 (+ 79us) server_negotiation.cc:407] Connection header received
0928 15:03:10.033348 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.033349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.033373 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.033474 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.034345 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.035371 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.036327 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.036627 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.036725 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.038470 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.038474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.038474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.038484 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.040896 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.041562 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.041566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.041566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.041681 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.041958 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.041959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.041960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.042110 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.042217 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.042410 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.042543 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:03:10.042656 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.033055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45455 (local address 127.25.123.1:52193)
0928 15:03:10.033191 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.033202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.033283 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.033589 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.033592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.033884 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.034176 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.034181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.035505 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.035507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.036175 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.036179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.036727 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.036728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.036799 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.037762 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.037771 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.038257 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.041053 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.041055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.041056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.041313 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.041777 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.041779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.041779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.041885 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.042196 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.042197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.042273 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.042380 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.042542 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab.
Entry for principal oryx/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab.
I20260928 15:03:10.073622 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45455
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab
--principal=oryx/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:10.181497 1420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:10.181778 1420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:10.181813 1420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:10.184087 1420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:10.184226 1420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:10.184258 1420 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:03:10.184283 1420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:10.187098 1420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab
--principal=oryx/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:10.187501 1420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:10.187955 1420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:10.196693 1426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:10.196844 1425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:10.196730 1428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:10.197675 1420 server_base.cc:1034] running on GCE node
I20260928 15:03:10.198096 1420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:10.198623 1420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:10.199831 1420 hybrid_clock.cc:648] HybridClock initialized: now 1790607790199825 us; error 65 us; skew 500 ppm
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.203193 1420 init.cc:377] Logged in from keytab as oryx/127.25.123.2@KRBTEST.COM (short username oryx)
I20260928 15:03:10.203754 1420 webserver.cc:460] Webserver started at http://127.25.123.2:35173/ using document root <none> and password file <none>
I20260928 15:03:10.204044 1420 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:10.204123 1420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:10.204290 1420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:10.205528 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "d4de4781f89143dea24a2b2a541d6737"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-qnz0"
server_key: "54dde1241904bb06c39d5172c55884ad"
server_key_iv: "74a70e96e3da71c77c210b175c90de29"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.205926 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "d4de4781f89143dea24a2b2a541d6737"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-qnz0"
server_key: "54dde1241904bb06c39d5172c55884ad"
server_key_iv: "74a70e96e3da71c77c210b175c90de29"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.208053 1420 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 15:03:10.209282 1435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.209686 1420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:10.210013 1420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "d4de4781f89143dea24a2b2a541d6737"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-qnz0"
server_key: "54dde1241904bb06c39d5172c55884ad"
server_key_iv: "74a70e96e3da71c77c210b175c90de29"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.210146 1420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:10.229481 1420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:10.232195 1420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:10.232342 1420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:10.233100 1420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:10.233172 1420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.233208 1420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:10.233237 1420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.241770 1420 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:40853
I20260928 15:03:10.241832 1551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:40853 every 8 connection(s)
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:10.242259 1420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:10.246879 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.233848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:45419 (local address 127.25.123.62:45455)
0928 15:03:10.233914 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.233917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.234145 (+ 228us) server_negotiation.cc:407] Connection header received
0928 15:03:10.234760 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.234761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.234776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.234855 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.235864 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.236863 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.239343 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.239699 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.239792 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.242847 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.242851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.242852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.242866 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.244978 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.245612 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.245615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.245615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.245688 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.246035 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.246036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.246037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.246217 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.246303 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.246649 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.246807 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:03:10.246965 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.233823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45455 (local address 127.25.123.2:45419)
0928 15:03:10.233999 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.234042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.234622 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.234983 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.234987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.235418 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.235733 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.235741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.237067 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.237070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.239204 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.239211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.239866 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.239868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.240049 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.240836 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.240856 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.242741 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.245138 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.245148 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.245150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.245487 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.245790 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.245791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.245791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.245910 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.246324 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.246330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.246475 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.246583 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.246749 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260928 15:03:10.247756 1552 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45455
I20260928 15:03:10.247867 1552 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:10.248099 1552 heartbeater.cc:511] Master 127.25.123.62:45455 requested a full tablet report, sending...
I20260928 15:03:10.248576 1205 ts_manager.cc:194] Registered new tserver with Master: d4de4781f89143dea24a2b2a541d6737 (127.25.123.2:40853)
I20260928 15:03:10.249240 1205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:45419
I20260928 15:03:10.252557 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 1420
I20260928 15:03:10.252689 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:10.252955 26092 external_mini_cluster.cc:1428] Setting key 7ef7cb0e332e912ce9b77b58ef72ae87
I20260928 15:03:10.257194 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.249137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45455 (local address 127.25.123.2:43063)
0928 15:03:10.249329 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.249342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.249461 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.249642 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.249644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.249873 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.249933 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.249935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.250855 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.250858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.251593 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.251607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.252158 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.252160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.252255 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.252858 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.252863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.253308 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.255400 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.255402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.255402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.255681 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.255914 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.255915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.255916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.255990 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.256700 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.256701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.256771 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.256871 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.257017 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260928 15:03:10.257236 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.249244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:43063 (local address 127.25.123.62:45455)
0928 15:03:10.249298 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.249301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.249443 (+ 142us) server_negotiation.cc:407] Connection header received
0928 15:03:10.249518 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.249518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.249530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.249591 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.250056 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.250556 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.251831 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.252063 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.252153 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.253409 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.253411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.253411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.253419 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.255282 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.255785 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.255786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.255787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.255844 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.256141 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.256143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.256144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.256584 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.256698 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.256896 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.257142 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
WARNING: no policy specified for oryx/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab.
Entry for principal oryx/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab.
I20260928 15:03:10.290266 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45455
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab
--principal=oryx/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:10.395013 1561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:10.395272 1561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:10.395301 1561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:10.396760 1561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:10.396821 1561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:10.396847 1561 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:03:10.396867 1561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:10.398949 1561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab
--principal=oryx/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.123.62:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:10.399238 1561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:10.399452 1561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:10.405628 1566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:10.405618 1569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:10.405618 1567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:10.405774 1561 server_base.cc:1034] running on GCE node
I20260928 15:03:10.406081 1561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:10.406404 1561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:10.407580 1561 hybrid_clock.cc:648] HybridClock initialized: now 1790607790407568 us; error 29 us; skew 500 ppm
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:10.411547 1561 init.cc:377] Logged in from keytab as oryx/127.25.123.3@KRBTEST.COM (short username oryx)
I20260928 15:03:10.412184 1561 webserver.cc:460] Webserver started at http://127.25.123.3:42085/ using document root <none> and password file <none>
I20260928 15:03:10.412492 1561 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:10.412546 1561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:10.412678 1561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:10.413875 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "ae815510be4243cf873bdf2315335f71"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-qnz0"
server_key: "4f1285877020c59bee77705d1d0783d5"
server_key_iv: "edecdd6fc1cc325ac8bdf33fcf7bd849"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.414289 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "ae815510be4243cf873bdf2315335f71"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-qnz0"
server_key: "4f1285877020c59bee77705d1d0783d5"
server_key_iv: "edecdd6fc1cc325ac8bdf33fcf7bd849"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.415802 1561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:03:10.416989 1576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.417271 1561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:03:10.417395 1561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "ae815510be4243cf873bdf2315335f71"
format_stamp: "Formatted at 2026-09-28 15:03:10 on dist-test-slave-qnz0"
server_key: "4f1285877020c59bee77705d1d0783d5"
server_key_iv: "edecdd6fc1cc325ac8bdf33fcf7bd849"
server_key_version: "encryptionkey@0"
I20260928 15:03:10.417487 1561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:10.436344 1561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:10.439796 1561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:10.440049 1561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:10.440927 1561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:10.440999 1561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.441048 1561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:10.441078 1561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:10.450400 1561 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:36837
I20260928 15:03:10.450532 1692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:36837 every 8 connection(s)
I20260928 15:03:10.450933 1561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:10.453059 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.441382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45455 (local address 127.25.123.3:53369)
0928 15:03:10.441558 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.441600 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.442166 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.442495 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.442499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.442829 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.443045 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.443049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.443846 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.443847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.444757 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.444759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.445229 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.445231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.445378 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.446006 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.446014 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.448487 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.451426 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.451431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.451434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.451738 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.452009 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.452009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.452010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.452114 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.452461 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.452468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.452630 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.452736 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.452884 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260928 15:03:10.453061 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.441363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:53369 (local address 127.25.123.62:45455)
0928 15:03:10.441456 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.441459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.441701 (+ 242us) server_negotiation.cc:407] Connection header received
0928 15:03:10.442330 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.442331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.442344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.442405 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.443153 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.443723 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.444870 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.445127 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.445217 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.448609 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.448611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.448612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.448629 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.450995 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.451838 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.451840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.451841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.451910 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.452174 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.452174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.452175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.452300 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.452370 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.452803 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.452984 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:03:10.454216 1693 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45455
I20260928 15:03:10.454386 1693 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:10.454792 1693 heartbeater.cc:511] Master 127.25.123.62:45455 requested a full tablet report, sending...
I20260928 15:03:10.455453 1205 ts_manager.cc:194] Registered new tserver with Master: ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837)
I20260928 15:03:10.456137 1205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:53369
I20260928 15:03:10.457449 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 1561
I20260928 15:03:10.457571 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:10.457820 26092 external_mini_cluster.cc:1428] Setting key 6538afad5a0aefb1c45d5a77372da9ff
I20260928 15:03:10.460078 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:03:10.466262 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.455516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45455 (local address 127.25.123.3:47405)
0928 15:03:10.455726 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.455740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.455832 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.456020 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.456021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.456329 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.456460 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.456464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.457464 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.457466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.458279 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.458282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.459292 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.459294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.459448 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.460531 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.460537 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.460922 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.463787 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.463788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.463789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.464336 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.464740 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.464743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.464743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.465002 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.465547 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.465548 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.465861 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.465942 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.466113 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260928 15:03:10.466276 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.455572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:47405 (local address 127.25.123.62:45455)
0928 15:03:10.455636 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.455639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.455885 (+ 246us) server_negotiation.cc:407] Connection header received
0928 15:03:10.455921 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.455922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.455933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.455999 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.456576 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.457323 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.458403 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.459078 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.459290 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.461074 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.461077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.461077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.461090 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.463619 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.464522 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.464526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.464526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.464608 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.465217 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.465219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.465219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.465432 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.465514 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.466019 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.466126 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:10.492372 1256 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:10.484493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.25.123.62:45455)
0928 15:03:10.484571 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.484576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.484591 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:03:10.484638 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.484639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.484661 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.484730 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.485568 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.486208 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.487547 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.487759 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.487829 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.492182 (+ 4353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38124: BlockingRecv error: recv got EOF from 127.0.0.1:38124 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:10.504557 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.493848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.25.123.62:45455)
0928 15:03:10.493946 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.493948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.494071 (+ 123us) server_negotiation.cc:407] Connection header received
0928 15:03:10.494106 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.494107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.494129 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.494206 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.494866 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.495463 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.497006 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.497290 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.497375 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.500986 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.500989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.500989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.501000 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.503048 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.503600 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.503602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.503602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.503681 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.503891 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.503892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.503892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.504032 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.504150 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.504306 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.504436 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:03:10.507036 1205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:10.509182 1205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:10.518659 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.513869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40853 (local address 127.0.0.1:46592)
0928 15:03:10.514227 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.514240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.514298 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.515090 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.515091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.515093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:10.515130 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.515131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.516546 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.516547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.517531 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.517533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.518380 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.518381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.518449 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.518455 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.518485 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.518554 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":38,"thread_start_us":110,"threads_started":1}
I20260928 15:03:10.518803 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.513837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40075 (local address 127.0.0.1:38328)
0928 15:03:10.514067 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.514117 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.514198 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.514709 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.514713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.514715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:10.514804 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.514806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.515887 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.515887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.517127 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.517130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.518298 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.518299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.518411 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.518431 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.518562 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.518622 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260928 15:03:10.519201 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.514091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36837 (local address 127.0.0.1:51260)
0928 15:03:10.514349 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.514361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.514408 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.514780 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.514781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.514783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:10.514914 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.514918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.516677 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.516678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.517887 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.517891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.518955 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.518956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.519038 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.519042 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.519091 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.519128 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":65,"threads_started":1}
I20260928 15:03:10.519217 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.513971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.25.123.2:40853)
0928 15:03:10.514904 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.514909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.514921 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:10.514965 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.514967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.514986 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.515086 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:10.515211 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.516452 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.517625 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.518276 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.518831 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.519026 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.519065 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.519113 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":96,"threads_started":1}
I20260928 15:03:10.519486 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.514302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.25.123.3:36837)
0928 15:03:10.514577 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.514583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.514595 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:10.514657 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.514659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.514678 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.514767 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:10.515024 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.516313 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.518006 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.518777 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.518845 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.518917 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.519334 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.519389 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":121,"threads_started":1}
I20260928 15:03:10.519733 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.514343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.25.123.1:40075)
0928 15:03:10.514511 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.514517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.514529 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:10.514577 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.514579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.514595 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.514703 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:10.514910 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.515784 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.517252 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.518170 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.518293 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.518400 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.518584 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.518655 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260928 15:03:10.520020 1486 tablet_service.cc:1467] Processing CreateTablet for tablet c5d72437c4164a1ab48aa07728b5b164 (DEFAULT_TABLE table=test-table [id=995122933a8e4d7f86f965d32a9df4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:10.520541 1486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d72437c4164a1ab48aa07728b5b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.520588 1345 tablet_service.cc:1467] Processing CreateTablet for tablet c5d72437c4164a1ab48aa07728b5b164 (DEFAULT_TABLE table=test-table [id=995122933a8e4d7f86f965d32a9df4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:10.520942 1345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d72437c4164a1ab48aa07728b5b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.521097 1627 tablet_service.cc:1467] Processing CreateTablet for tablet c5d72437c4164a1ab48aa07728b5b164 (DEFAULT_TABLE table=test-table [id=995122933a8e4d7f86f965d32a9df4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:10.521502 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d72437c4164a1ab48aa07728b5b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.524808 1721 tablet_bootstrap.cc:492] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Bootstrap starting.
I20260928 15:03:10.525225 1722 tablet_bootstrap.cc:492] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Bootstrap starting.
I20260928 15:03:10.525976 1721 tablet_bootstrap.cc:654] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.525976 1722 tablet_bootstrap.cc:654] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.526358 1722 log.cc:824] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:10.526391 1721 log.cc:824] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:10.527060 1723 tablet_bootstrap.cc:492] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Bootstrap starting.
I20260928 15:03:10.527558 1722 tablet_bootstrap.cc:492] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: No bootstrap required, opened a new log
I20260928 15:03:10.527696 1722 ts_tablet_manager.cc:1397] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:10.527863 1721 tablet_bootstrap.cc:492] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: No bootstrap required, opened a new log
I20260928 15:03:10.527920 1723 tablet_bootstrap.cc:654] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.527981 1721 ts_tablet_manager.cc:1397] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:10.528335 1723 log.cc:824] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:10.529306 1723 tablet_bootstrap.cc:492] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: No bootstrap required, opened a new log
I20260928 15:03:10.529381 1723 ts_tablet_manager.cc:1397] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:10.530171 1721 raft_consensus.cc:348] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.530354 1721 raft_consensus.cc:374] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.530383 1721 raft_consensus.cc:729] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d66b08cc2f45e58aee35fddaef3811, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.530257 1722 raft_consensus.cc:348] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.530435 1722 raft_consensus.cc:374] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.530470 1722 raft_consensus.cc:729] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae815510be4243cf873bdf2315335f71, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.530548 1721 consensus_queue.cc:260] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.530597 1722 consensus_queue.cc:260] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.530907 1722 ts_tablet_manager.cc:1428] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 15:03:10.530907 1721 ts_tablet_manager.cc:1428] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:10.531142 1411 heartbeater.cc:503] Master 127.25.123.62:45455 was elected leader, sending a full tablet report...
I20260928 15:03:10.531663 1723 raft_consensus.cc:348] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.531834 1723 raft_consensus.cc:374] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.531859 1723 raft_consensus.cc:729] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.531993 1723 consensus_queue.cc:260] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.532284 1723 ts_tablet_manager.cc:1428] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:10.532357 1552 heartbeater.cc:503] Master 127.25.123.62:45455 was elected leader, sending a full tablet report...
I20260928 15:03:10.532469 1693 heartbeater.cc:503] Master 127.25.123.62:45455 was elected leader, sending a full tablet report...
I20260928 15:03:10.606146 1728 raft_consensus.cc:482] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:10.606295 1728 raft_consensus.cc:504] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.606688 1728 leader_election.cc:288] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4de4781f89143dea24a2b2a541d6737 (127.25.123.2:40853), c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075)
I20260928 15:03:10.609939 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.607057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:59033 (local address 127.25.123.2:40853)
0928 15:03:10.607102 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.607104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.607116 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:10.607264 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.607265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.607277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.607354 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:10.607514 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.608001 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.609058 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.609441 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.609609 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.609621 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.609816 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.609858 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:03:10.609954 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.607001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40853 (local address 127.25.123.3:59033)
0928 15:03:10.607067 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.607081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.607130 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.607359 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.607360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.607362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:10.607441 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.607443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.608104 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.608105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.608954 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.608957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.609609 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.609610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.609724 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.609728 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.609808 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.609855 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 15:03:10.610020 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.607061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:54083 (local address 127.25.123.1:40075)
0928 15:03:10.607130 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.607132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.607288 (+ 156us) server_negotiation.cc:407] Connection header received
0928 15:03:10.607363 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.607364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.607375 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.607429 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:10.607596 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.608168 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.609210 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.609624 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.609716 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.609726 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.609903 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.609945 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:03:10.610065 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.607017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40075 (local address 127.25.123.3:54083)
0928 15:03:10.607194 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.607204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.607294 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.607444 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.607445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.607446 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:10.607519 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.607520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.608294 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.608295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.609113 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.609115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.609729 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.609731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.609823 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.609827 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.609892 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.609934 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260928 15:03:10.610396 1365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d72437c4164a1ab48aa07728b5b164" candidate_uuid: "ae815510be4243cf873bdf2315335f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811" is_pre_election: true
I20260928 15:03:10.610479 1506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d72437c4164a1ab48aa07728b5b164" candidate_uuid: "ae815510be4243cf873bdf2315335f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4de4781f89143dea24a2b2a541d6737" is_pre_election: true
I20260928 15:03:10.610597 1365 raft_consensus.cc:2454] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae815510be4243cf873bdf2315335f71 in term 0.
I20260928 15:03:10.610669 1506 raft_consensus.cc:2454] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae815510be4243cf873bdf2315335f71 in term 0.
I20260928 15:03:10.610836 1579 leader_election.cc:302] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, c6d66b08cc2f45e58aee35fddaef3811; no voters:
I20260928 15:03:10.610991 1728 raft_consensus.cc:2781] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:10.611074 1728 raft_consensus.cc:482] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:10.611110 1728 raft_consensus.cc:3037] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:10.612069 1728 raft_consensus.cc:504] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.612259 1728 leader_election.cc:288] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [CANDIDATE]: Term 1 election: Requested vote from peers d4de4781f89143dea24a2b2a541d6737 (127.25.123.2:40853), c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075)
I20260928 15:03:10.612574 1506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d72437c4164a1ab48aa07728b5b164" candidate_uuid: "ae815510be4243cf873bdf2315335f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4de4781f89143dea24a2b2a541d6737"
I20260928 15:03:10.612669 1506 raft_consensus.cc:3037] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:10.612757 1365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d72437c4164a1ab48aa07728b5b164" candidate_uuid: "ae815510be4243cf873bdf2315335f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811"
I20260928 15:03:10.612851 1365 raft_consensus.cc:3037] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:10.613692 1506 raft_consensus.cc:2454] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae815510be4243cf873bdf2315335f71 in term 1.
I20260928 15:03:10.613775 1365 raft_consensus.cc:2454] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae815510be4243cf873bdf2315335f71 in term 1.
I20260928 15:03:10.614061 1579 leader_election.cc:302] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, c6d66b08cc2f45e58aee35fddaef3811; no voters:
I20260928 15:03:10.614169 1728 raft_consensus.cc:2781] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:10.614672 1728 raft_consensus.cc:686] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 1 LEADER]: Becoming Leader. State: Replica: ae815510be4243cf873bdf2315335f71, State: Running, Role: LEADER
I20260928 15:03:10.615152 1728 consensus_queue.cc:237] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.616807 1205 catalog_manager.cc:5495] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae815510be4243cf873bdf2315335f71 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "ae815510be4243cf873bdf2315335f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260928 15:03:10.639812 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.637282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.25.123.3:36837)
0928 15:03:10.637360 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.637362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.637369 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:03:10.637405 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.637406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.637425 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.637508 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:10.637669 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.638200 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.638975 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.639167 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.639246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.639449 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:10.639558 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:10.639656 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.639676 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.639699 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 15:03:10.642104 1506 raft_consensus.cc:1261] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Refusing update from remote peer ae815510be4243cf873bdf2315335f71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:10.642218 1365 raft_consensus.cc:1261] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Refusing update from remote peer ae815510be4243cf873bdf2315335f71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:10.642679 1728 consensus_queue.cc:1035] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:10.642995 1731 consensus_queue.cc:1035] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:10.646503 1735 mvcc.cc:205] Tried to move back new op lower bound from 7334329510467031040 to 7334329510360768512. Current Snapshot: MvccSnapshot[applied={T|T < 7334329510467031040}]
I20260928 15:03:10.648635 1205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:10.648772 1205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:10.649721 1205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Sending DeleteTablet for 3 replicas of tablet c5d72437c4164a1ab48aa07728b5b164
I20260928 15:03:10.650074 1486 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d72437c4164a1ab48aa07728b5b164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:10 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:46592
I20260928 15:03:10.650173 1345 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d72437c4164a1ab48aa07728b5b164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:10 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:38328
I20260928 15:03:10.650213 1627 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d72437c4164a1ab48aa07728b5b164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:10 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:51260
I20260928 15:03:10.650364 1205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:03:10.650493 1746 tablet_replica.cc:330] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: stopping tablet replica
W20260928 15:03:10.650521 1205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:10.650686 1746 raft_consensus.cc:2229] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:10.650827 1746 raft_consensus.cc:2258] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:10.651588 1746 ts_tablet_manager.cc:1905] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:10.652209 1486 tablet_service.cc:1467] Processing CreateTablet for tablet 51bf04a87b3746329f90037f6841ed26 (DEFAULT_TABLE table=test-table [id=7644758c8e3c4b168d23fad8f66c2ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:10.652352 1486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bf04a87b3746329f90037f6841ed26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.652379 1627 tablet_service.cc:1467] Processing CreateTablet for tablet 51bf04a87b3746329f90037f6841ed26 (DEFAULT_TABLE table=test-table [id=7644758c8e3c4b168d23fad8f66c2ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:10.652529 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bf04a87b3746329f90037f6841ed26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.652382 1345 tablet_service.cc:1467] Processing CreateTablet for tablet 51bf04a87b3746329f90037f6841ed26 (DEFAULT_TABLE table=test-table [id=7644758c8e3c4b168d23fad8f66c2ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:10.652711 1345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bf04a87b3746329f90037f6841ed26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.653909 1746 ts_tablet_manager.cc:1918] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:10.653990 1746 log.cc:1196] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/c5d72437c4164a1ab48aa07728b5b164
I20260928 15:03:10.654261 1746 ts_tablet_manager.cc:1939] T c5d72437c4164a1ab48aa07728b5b164 P ae815510be4243cf873bdf2315335f71: Deleting consensus metadata
I20260928 15:03:10.654675 1723 tablet_bootstrap.cc:492] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Bootstrap starting.
I20260928 15:03:10.655141 1723 tablet_bootstrap.cc:654] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.655139 1721 tablet_bootstrap.cc:492] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Bootstrap starting.
I20260928 15:03:10.655524 1722 tablet_bootstrap.cc:492] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Bootstrap starting.
I20260928 15:03:10.655566 1721 tablet_bootstrap.cc:654] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.655956 1722 tablet_bootstrap.cc:654] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.651911 1744 tablet_replica.cc:330] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: stopping tablet replica
I20260928 15:03:10.656330 1744 raft_consensus.cc:2229] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:10.656715 1744 raft_consensus.cc:2258] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:10.656692 1191 catalog_manager.cc:4841] TS ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837): tablet c5d72437c4164a1ab48aa07728b5b164 (table test-table [id=995122933a8e4d7f86f965d32a9df4dd]) successfully deleted
I20260928 15:03:10.656924 1722 tablet_bootstrap.cc:492] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: No bootstrap required, opened a new log
I20260928 15:03:10.656975 1722 ts_tablet_manager.cc:1397] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:10.657133 1722 raft_consensus.cc:348] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.657220 1722 raft_consensus.cc:374] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.657244 1722 raft_consensus.cc:729] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae815510be4243cf873bdf2315335f71, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.657263 1744 ts_tablet_manager.cc:1905] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:10.657298 1722 consensus_queue.cc:260] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.657430 1722 ts_tablet_manager.cc:1428] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:10.659482 1744 ts_tablet_manager.cc:1918] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:10.659585 1744 log.cc:1196] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/c5d72437c4164a1ab48aa07728b5b164
I20260928 15:03:10.659593 1721 tablet_bootstrap.cc:492] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: No bootstrap required, opened a new log
I20260928 15:03:10.659648 1721 ts_tablet_manager.cc:1397] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 15:03:10.659765 1723 tablet_bootstrap.cc:492] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: No bootstrap required, opened a new log
I20260928 15:03:10.659822 1723 ts_tablet_manager.cc:1397] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 15:03:10.659899 1744 ts_tablet_manager.cc:1939] T c5d72437c4164a1ab48aa07728b5b164 P d4de4781f89143dea24a2b2a541d6737: Deleting consensus metadata
I20260928 15:03:10.659852 1721 raft_consensus.cc:348] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.659938 1721 raft_consensus.cc:374] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.659966 1721 raft_consensus.cc:729] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d66b08cc2f45e58aee35fddaef3811, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.659979 1723 raft_consensus.cc:348] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.660024 1721 consensus_queue.cc:260] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.660050 1723 raft_consensus.cc:374] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.660076 1723 raft_consensus.cc:729] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.660137 1721 ts_tablet_manager.cc:1428] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:10.660130 1723 consensus_queue.cc:260] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.660239 1723 ts_tablet_manager.cc:1428] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:10.660787 1189 catalog_manager.cc:4841] TS d4de4781f89143dea24a2b2a541d6737 (127.25.123.2:40853): tablet c5d72437c4164a1ab48aa07728b5b164 (table test-table [id=995122933a8e4d7f86f965d32a9df4dd]) successfully deleted
I20260928 15:03:10.664676 1745 tablet_replica.cc:330] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: stopping tablet replica
I20260928 15:03:10.665091 1745 raft_consensus.cc:2229] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:10.665267 1745 raft_consensus.cc:2258] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:10.666119 1745 ts_tablet_manager.cc:1905] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:10.668334 1745 ts_tablet_manager.cc:1918] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:10.668486 1745 log.cc:1196] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/c5d72437c4164a1ab48aa07728b5b164
I20260928 15:03:10.669157 1745 ts_tablet_manager.cc:1939] T c5d72437c4164a1ab48aa07728b5b164 P c6d66b08cc2f45e58aee35fddaef3811: Deleting consensus metadata
I20260928 15:03:10.669870 1191 catalog_manager.cc:4841] TS c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075): tablet c5d72437c4164a1ab48aa07728b5b164 (table test-table [id=995122933a8e4d7f86f965d32a9df4dd]) successfully deleted
W20260928 15:03:10.701712 1694 tablet.cc:2369] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:10.725941 1729 raft_consensus.cc:482] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:10.726054 1729 raft_consensus.cc:504] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.726433 1729 leader_election.cc:288] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075), ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837)
I20260928 15:03:10.730574 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.726789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36837 (local address 127.25.123.2:35531)
0928 15:03:10.727109 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.727120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.727220 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.727449 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.727450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.727453 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:10.727523 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.727524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.728382 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.728383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.729393 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.729396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.730227 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.730228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.730316 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.730322 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.730396 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.730456 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":90,"threads_started":1}
I20260928 15:03:10.730566 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.726986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:35531 (local address 127.25.123.3:36837)
0928 15:03:10.727036 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.727038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.727187 (+ 149us) server_negotiation.cc:407] Connection header received
0928 15:03:10.727309 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.727311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.727334 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.727387 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:10.727614 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.728306 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.729512 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.729988 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.730214 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.730236 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.730416 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.730488 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:03:10.730841 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bf04a87b3746329f90037f6841ed26" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71" is_pre_election: true
I20260928 15:03:10.731005 1647 raft_consensus.cc:2454] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:10.731106 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.726677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40075 (local address 127.25.123.2:48853)
0928 15:03:10.726807 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.726827 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.726941 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.727169 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.727171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.727175 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:10.727262 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.727264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.728376 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.728377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.729755 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.729759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.730812 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.730813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.730928 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.730931 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.730997 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.731041 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 15:03:10.731168 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.726918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:48853 (local address 127.25.123.1:40075)
0928 15:03:10.726980 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.726983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.726995 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:10.727033 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.727034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.727044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.727118 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:10.727392 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.728236 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.729902 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.730623 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.730716 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.730728 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.731030 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.731083 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:03:10.731241 1438 leader_election.cc:302] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:10.731469 1729 raft_consensus.cc:2781] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:10.731523 1729 raft_consensus.cc:482] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:10.731546 1729 raft_consensus.cc:3037] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:10.731570 1365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bf04a87b3746329f90037f6841ed26" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811" is_pre_election: true
I20260928 15:03:10.731665 1365 raft_consensus.cc:2454] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:10.732717 1729 raft_consensus.cc:504] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.733108 1729 leader_election.cc:288] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Requested vote from peers c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075), ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837)
I20260928 15:03:10.733652 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bf04a87b3746329f90037f6841ed26" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71"
I20260928 15:03:10.733762 1647 raft_consensus.cc:3037] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:10.734558 1647 raft_consensus.cc:2454] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:10.734805 1365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bf04a87b3746329f90037f6841ed26" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811"
I20260928 15:03:10.734929 1365 raft_consensus.cc:3037] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:10.735038 1438 leader_election.cc:302] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:10.735157 1729 raft_consensus.cc:2781] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:10.735716 1729 raft_consensus.cc:686] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 1 LEADER]: Becoming Leader. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Running, Role: LEADER
I20260928 15:03:10.735756 1365 raft_consensus.cc:2454] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:10.735843 1729 consensus_queue.cc:237] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:10.737146 1204 catalog_manager.cc:5495] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4de4781f89143dea24a2b2a541d6737 (127.25.123.2). New cstate: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } health_report { overall_health: UNKNOWN } } }
W20260928 15:03:10.743393 1553 tablet.cc:2369] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:03:10.779608 1412 tablet.cc:2369] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:03:10 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607789, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.62@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:10.791920 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.781787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.25.123.62:45455)
0928 15:03:10.781860 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:10.781863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:10.781987 (+ 124us) server_negotiation.cc:407] Connection header received
0928 15:03:10.782120 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:10.782121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:10.782139 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:10.782230 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:10.782760 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.783546 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.784528 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.784778 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.784857 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.788313 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:10.788315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:10.788315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:10.788325 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:10.790344 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.790856 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.790858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.790858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.790926 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:10.791200 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:10.791201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:10.791201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:10.791366 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:10.791443 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:10.791702 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:03:10.791793 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:03:10.792291 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:10.781488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45455 (local address 127.0.0.1:38140)
0928 15:03:10.781894 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:10.781913 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:10.782010 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:10.782242 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:10.782243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:10.782579 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:10.782629 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.782632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.783745 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.783748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.784349 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:10.784351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:10.784908 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:10.784911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:10.785026 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:10.786241 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:10.786248 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:10.788182 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:10.790507 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.790519 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.790521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.790757 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.791042 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:10.791043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:10.791043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:10.791116 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:10.791477 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:10.791489 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:10.791567 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 15:03:10.791668 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:03:10.791909 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"thread_start_us":264,"threads_started":1}
I20260928 15:03:10.796716 1204 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:38140:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:03:10.797087 1204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:10.799437 1486 tablet_service.cc:1467] Processing CreateTablet for tablet 1c52b13e2041493d8544778c09a7a577 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f479f83905204383a75f5aad3ee7e034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:03:10.799463 1345 tablet_service.cc:1467] Processing CreateTablet for tablet 1c52b13e2041493d8544778c09a7a577 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f479f83905204383a75f5aad3ee7e034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:03:10.799651 1345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c52b13e2041493d8544778c09a7a577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.799654 1486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c52b13e2041493d8544778c09a7a577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.799659 1627 tablet_service.cc:1467] Processing CreateTablet for tablet 1c52b13e2041493d8544778c09a7a577 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f479f83905204383a75f5aad3ee7e034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:03:10.799856 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c52b13e2041493d8544778c09a7a577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:10.801689 1721 tablet_bootstrap.cc:492] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: Bootstrap starting.
I20260928 15:03:10.801651 1723 tablet_bootstrap.cc:492] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: Bootstrap starting.
I20260928 15:03:10.801791 1722 tablet_bootstrap.cc:492] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: Bootstrap starting.
I20260928 15:03:10.802476 1723 tablet_bootstrap.cc:654] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.802457 1721 tablet_bootstrap.cc:654] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.802578 1722 tablet_bootstrap.cc:654] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:10.803490 1721 tablet_bootstrap.cc:492] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: No bootstrap required, opened a new log
I20260928 15:03:10.803490 1723 tablet_bootstrap.cc:492] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: No bootstrap required, opened a new log
I20260928 15:03:10.803547 1721 ts_tablet_manager.cc:1397] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:10.803547 1723 ts_tablet_manager.cc:1397] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:10.803723 1721 raft_consensus.cc:348] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:10.803719 1723 raft_consensus.cc:348] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:10.803812 1723 raft_consensus.cc:374] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.803833 1723 raft_consensus.cc:729] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.803812 1721 raft_consensus.cc:374] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.803908 1721 raft_consensus.cc:729] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d66b08cc2f45e58aee35fddaef3811, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.803879 1723 consensus_queue.cc:260] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:10.803953 1721 consensus_queue.cc:260] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:10.803985 1723 ts_tablet_manager.cc:1428] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:10.804044 1721 ts_tablet_manager.cc:1428] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:10.804149 1729 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:10.804217 1727 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:10.804263 1729 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:10.804291 1727 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:10.805073 1722 tablet_bootstrap.cc:492] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: No bootstrap required, opened a new log
I20260928 15:03:10.805173 1722 ts_tablet_manager.cc:1397] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:10.805343 1722 raft_consensus.cc:348] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:10.805441 1722 raft_consensus.cc:374] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:10.805462 1722 raft_consensus.cc:729] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae815510be4243cf873bdf2315335f71, State: Initialized, Role: FOLLOWER
I20260928 15:03:10.805516 1722 consensus_queue.cc:260] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:10.805624 1722 ts_tablet_manager.cc:1428] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:10.805845 1739 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:10.806005 1739 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:11.012315 1729 raft_consensus.cc:482] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:11.012471 1729 raft_consensus.cc:504] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:11.012902 1729 leader_election.cc:288] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837), c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075)
I20260928 15:03:11.013262 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c52b13e2041493d8544778c09a7a577" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71" is_pre_election: true
I20260928 15:03:11.013443 1647 raft_consensus.cc:2454] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:11.013428 1365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c52b13e2041493d8544778c09a7a577" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811" is_pre_election: true
I20260928 15:03:11.013527 1365 raft_consensus.cc:2454] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:11.013775 1438 leader_election.cc:302] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:11.013908 1729 raft_consensus.cc:2781] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:11.013958 1729 raft_consensus.cc:482] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:11.013984 1729 raft_consensus.cc:3037] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.015138 1729 raft_consensus.cc:504] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:11.015355 1729 leader_election.cc:288] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Requested vote from peers ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837), c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075)
I20260928 15:03:11.015566 1365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c52b13e2041493d8544778c09a7a577" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811"
I20260928 15:03:11.015576 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c52b13e2041493d8544778c09a7a577" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71"
I20260928 15:03:11.015810 1365 raft_consensus.cc:3037] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.015847 1647 raft_consensus.cc:3037] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.016760 1647 raft_consensus.cc:2454] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:11.016990 1438 leader_election.cc:302] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:11.017117 1729 raft_consensus.cc:2781] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:11.017196 1729 raft_consensus.cc:686] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [term 1 LEADER]: Becoming Leader. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Running, Role: LEADER
I20260928 15:03:11.017253 1729 consensus_queue.cc:237] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } }
I20260928 15:03:11.017361 1365 raft_consensus.cc:2454] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:11.017509 1751 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: TxnStatusTablet state changed. Reason: New leader d4de4781f89143dea24a2b2a541d6737. Latest consensus state: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:11.017606 1751 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:03:11.017819 1763 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:03:11.017956 1763 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:03:11.017997 1205 catalog_manager.cc:5495] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4de4781f89143dea24a2b2a541d6737 (127.25.123.2). New cstate: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:11.046417 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:11.042744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40853 (local address 127.0.0.1:46596)
0928 15:03:11.042847 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:11.042864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:11.042929 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:11.043398 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:11.043399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:11.043401 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:03:11.043454 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:11.043457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.044504 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.044506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:11.045229 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:11.045232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.045758 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.045759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:11.045819 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.045843 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:03:11.046198 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:03:11.046208 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:03:11.046289 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:03:11.046349 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 15:03:11.046417 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:11.042900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.25.123.2:40853)
0928 15:03:11.042989 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:11.042992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:11.043006 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:03:11.043231 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:11.043233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:11.043259 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:11.043364 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:11.043545 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.044380 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.045324 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.045656 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.045741 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.045954 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:11.046106 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:11.046196 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:11.046297 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 15:03:11.046323 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:03:11.048317 1365 raft_consensus.cc:1261] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.048491 1647 raft_consensus.cc:1261] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.048887 1751 consensus_queue.cc:1035] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.049218 1751 consensus_queue.cc:1035] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.049708 1727 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: TxnStatusTablet state changed. Reason: New leader d4de4781f89143dea24a2b2a541d6737. Latest consensus state: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:11.049816 1727 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:11.050383 1739 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: TxnStatusTablet state changed. Reason: New leader d4de4781f89143dea24a2b2a541d6737. Latest consensus state: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:11.050648 1739 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:11.051597 1736 mvcc.cc:205] Tried to move back new op lower bound from 7334329512130461696 to 7334329512006909952. Current Snapshot: MvccSnapshot[applied={T|T < 7334329512130461696}]
I20260928 15:03:11.051679 1729 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:11.051767 1729 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:03:11.052246 1739 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:11.052359 1739 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P ae815510be4243cf873bdf2315335f71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:11.052289 1734 mvcc.cc:205] Tried to move back new op lower bound from 7334329512130461696 to 7334329512006909952. Current Snapshot: MvccSnapshot[applied={T|T < 7334329512130461696}]
I20260928 15:03:11.053078 1729 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:11.053159 1729 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P d4de4781f89143dea24a2b2a541d6737: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:03:11.053880 1727 tablet_replica.cc:439] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } } peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } }
I20260928 15:03:11.053984 1727 tablet_replica.cc:442] T 1c52b13e2041493d8544778c09a7a577 P c6d66b08cc2f45e58aee35fddaef3811: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:03:11.057262 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:11.053816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.25.123.2:40853)
0928 15:03:11.054013 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:11.054015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:11.054023 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:11.054043 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:11.054044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:11.054055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:11.054099 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:11.054296 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.055076 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.056256 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.056599 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.056683 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.056946 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:11.056984 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:11.057040 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:11.057157 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 15:03:11.057179 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260928 15:03:11.064177 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:11.060827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:46103 (local address 127.25.123.2:40853)
0928 15:03:11.060904 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:11.060907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:11.061003 (+ 96us) server_negotiation.cc:407] Connection header received
0928 15:03:11.061033 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:11.061034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:11.061044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:11.061106 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:11.061344 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.062092 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.063116 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.063433 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.063513 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.063795 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:11.063835 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:11.063895 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:11.064027 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 15:03:11.064048 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:03:11.064184 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:11.060720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40853 (local address 127.25.123.2:46103)
0928 15:03:11.060919 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:11.060930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:11.060997 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:11.061204 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:11.061205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:11.061207 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:03:11.061265 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:11.061267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.062340 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.062341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:11.062993 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:11.062995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.063552 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.063553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:11.063611 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.063629 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:03:11.063947 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:03:11.063950 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:03:11.064001 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 15:03:11.064044 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260928 15:03:11.067679 1365 raft_consensus.cc:1261] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.067710 1647 raft_consensus.cc:1261] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.067971 1769 consensus_queue.cc:1035] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.068092 1751 consensus_queue.cc:1035] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.085742 1205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:11.085963 1205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:11.086843 1205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Sending DeleteTablet for 3 replicas of tablet 51bf04a87b3746329f90037f6841ed26
I20260928 15:03:11.087168 1345 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bf04a87b3746329f90037f6841ed26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:38328
I20260928 15:03:11.087173 1627 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bf04a87b3746329f90037f6841ed26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:51260
I20260928 15:03:11.087311 1745 tablet_replica.cc:330] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: stopping tablet replica
I20260928 15:03:11.087312 1746 tablet_replica.cc:330] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: stopping tablet replica
I20260928 15:03:11.087363 1745 raft_consensus.cc:2229] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:11.087370 1746 raft_consensus.cc:2229] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:11.087400 1745 raft_consensus.cc:2258] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.087409 1746 raft_consensus.cc:2258] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.087626 1485 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bf04a87b3746329f90037f6841ed26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:46592
I20260928 15:03:11.087738 1744 tablet_replica.cc:330] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: stopping tablet replica
I20260928 15:03:11.087796 1744 raft_consensus.cc:2229] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:11.087908 1744 raft_consensus.cc:2258] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.088094 1746 ts_tablet_manager.cc:1905] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.088191 1745 ts_tablet_manager.cc:1905] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.090250 1744 ts_tablet_manager.cc:1905] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.090994 1746 ts_tablet_manager.cc:1918] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:11.091059 1746 log.cc:1196] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/51bf04a87b3746329f90037f6841ed26
I20260928 15:03:11.091302 1746 ts_tablet_manager.cc:1939] T 51bf04a87b3746329f90037f6841ed26 P ae815510be4243cf873bdf2315335f71: Deleting consensus metadata
I20260928 15:03:11.091706 1191 catalog_manager.cc:4841] TS ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837): tablet 51bf04a87b3746329f90037f6841ed26 (table test-table [id=7644758c8e3c4b168d23fad8f66c2ed3]) successfully deleted
I20260928 15:03:11.094559 1745 ts_tablet_manager.cc:1918] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:11.094676 1745 log.cc:1196] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/51bf04a87b3746329f90037f6841ed26
I20260928 15:03:11.094949 1745 ts_tablet_manager.cc:1939] T 51bf04a87b3746329f90037f6841ed26 P c6d66b08cc2f45e58aee35fddaef3811: Deleting consensus metadata
I20260928 15:03:11.095463 1191 catalog_manager.cc:4841] TS c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075): tablet 51bf04a87b3746329f90037f6841ed26 (table test-table [id=7644758c8e3c4b168d23fad8f66c2ed3]) successfully deleted
I20260928 15:03:11.095897 1744 ts_tablet_manager.cc:1918] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:11.095988 1744 log.cc:1196] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/51bf04a87b3746329f90037f6841ed26
I20260928 15:03:11.096293 1744 ts_tablet_manager.cc:1939] T 51bf04a87b3746329f90037f6841ed26 P d4de4781f89143dea24a2b2a541d6737: Deleting consensus metadata
I20260928 15:03:11.096753 1189 catalog_manager.cc:4841] TS d4de4781f89143dea24a2b2a541d6737 (127.25.123.2:40853): tablet 51bf04a87b3746329f90037f6841ed26 (table test-table [id=7644758c8e3c4b168d23fad8f66c2ed3]) successfully deleted
Sep 28 15:03:11 dist-test-slave-qnz0 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:03:11 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:11 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:11.126892 1256 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:11.117564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.25.123.62:45455)
0928 15:03:11.117628 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:11.117631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:11.118079 (+ 448us) server_negotiation.cc:407] Connection header received
0928 15:03:11.118211 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:11.118212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:11.118230 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:11.118314 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:11.119389 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.120601 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.122175 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.122681 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.122807 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.126797 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38148: BlockingRecv error: recv got EOF from 127.0.0.1:38148 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 15:03:11 dist-test-slave-qnz0 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM
I20260928 15:03:11.138478 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:11.128809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.25.123.62:45455)
0928 15:03:11.128862 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:11.128864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:11.128989 (+ 125us) server_negotiation.cc:407] Connection header received
0928 15:03:11.129077 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:11.129078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:11.129088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:11.129165 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:11.129709 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.130364 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.131599 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.131941 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.131997 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.134765 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:11.134767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:11.134768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:11.134777 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:11.136329 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:11.136981 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:11.136984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:11.136985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:11.137117 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:11.137756 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:11.137758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:11.137759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:11.137914 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:11.138017 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:11.138325 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 15:03:11.138408 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:03:11.140328 1205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:11.140595 1205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:11.142035 1627 tablet_service.cc:1467] Processing CreateTablet for tablet ddc276c11294424eb0404fd24617f65a (DEFAULT_TABLE table=test-table [id=e4d0216f69c640c6b28861c558d54c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:11.142053 1485 tablet_service.cc:1467] Processing CreateTablet for tablet ddc276c11294424eb0404fd24617f65a (DEFAULT_TABLE table=test-table [id=e4d0216f69c640c6b28861c558d54c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:11.142035 1345 tablet_service.cc:1467] Processing CreateTablet for tablet ddc276c11294424eb0404fd24617f65a (DEFAULT_TABLE table=test-table [id=e4d0216f69c640c6b28861c558d54c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:11.142194 1485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc276c11294424eb0404fd24617f65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:11.142204 1345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc276c11294424eb0404fd24617f65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:11.142194 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc276c11294424eb0404fd24617f65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:11.143687 1721 tablet_bootstrap.cc:492] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: Bootstrap starting.
I20260928 15:03:11.144179 1721 tablet_bootstrap.cc:654] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:11.144380 1722 tablet_bootstrap.cc:492] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: Bootstrap starting.
I20260928 15:03:11.144649 1723 tablet_bootstrap.cc:492] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: Bootstrap starting.
I20260928 15:03:11.144832 1722 tablet_bootstrap.cc:654] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:11.145401 1723 tablet_bootstrap.cc:654] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:11.145985 1722 tablet_bootstrap.cc:492] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: No bootstrap required, opened a new log
I20260928 15:03:11.146054 1722 ts_tablet_manager.cc:1397] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:11.146148 1721 tablet_bootstrap.cc:492] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: No bootstrap required, opened a new log
I20260928 15:03:11.146219 1721 ts_tablet_manager.cc:1397] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 15:03:11.146189 1722 raft_consensus.cc:348] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.146272 1722 raft_consensus.cc:374] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:11.146365 1722 raft_consensus.cc:729] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae815510be4243cf873bdf2315335f71, State: Initialized, Role: FOLLOWER
I20260928 15:03:11.146416 1723 tablet_bootstrap.cc:492] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: No bootstrap required, opened a new log
I20260928 15:03:11.146446 1722 consensus_queue.cc:260] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.146665 1722 ts_tablet_manager.cc:1428] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:11.146442 1721 raft_consensus.cc:348] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.146530 1723 ts_tablet_manager.cc:1397] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:11.146938 1721 raft_consensus.cc:374] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:11.146981 1721 raft_consensus.cc:729] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d66b08cc2f45e58aee35fddaef3811, State: Initialized, Role: FOLLOWER
I20260928 15:03:11.147046 1721 consensus_queue.cc:260] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.147154 1721 ts_tablet_manager.cc:1428] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 15:03:11.147143 1723 raft_consensus.cc:348] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.147234 1723 raft_consensus.cc:374] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:11.147261 1723 raft_consensus.cc:729] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Initialized, Role: FOLLOWER
I20260928 15:03:11.147305 1723 consensus_queue.cc:260] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.147406 1723 ts_tablet_manager.cc:1428] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:11.435154 1777 raft_consensus.cc:482] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:11.435420 1777 raft_consensus.cc:504] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.435637 1777 leader_election.cc:288] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075), ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837)
I20260928 15:03:11.435885 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc276c11294424eb0404fd24617f65a" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811" is_pre_election: true
I20260928 15:03:11.435918 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc276c11294424eb0404fd24617f65a" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71" is_pre_election: true
I20260928 15:03:11.435999 1647 raft_consensus.cc:2454] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:11.436009 1364 raft_consensus.cc:2454] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:11.436230 1438 leader_election.cc:302] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:11.436362 1777 raft_consensus.cc:2781] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:11.436434 1777 raft_consensus.cc:482] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:11.436456 1777 raft_consensus.cc:3037] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.437273 1777 raft_consensus.cc:504] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.437410 1777 leader_election.cc:288] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Requested vote from peers c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075), ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837)
I20260928 15:03:11.437572 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc276c11294424eb0404fd24617f65a" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811"
I20260928 15:03:11.437659 1364 raft_consensus.cc:3037] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.437610 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc276c11294424eb0404fd24617f65a" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71"
I20260928 15:03:11.437693 1647 raft_consensus.cc:3037] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.438401 1364 raft_consensus.cc:2454] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:11.438583 1438 leader_election.cc:302] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6d66b08cc2f45e58aee35fddaef3811, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:11.438664 1647 raft_consensus.cc:2454] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:11.438695 1777 raft_consensus.cc:2781] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:11.438740 1777 raft_consensus.cc:686] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 1 LEADER]: Becoming Leader. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Running, Role: LEADER
I20260928 15:03:11.438794 1777 consensus_queue.cc:237] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.440752 1206 catalog_manager.cc:5495] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4de4781f89143dea24a2b2a541d6737 (127.25.123.2). New cstate: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:11.511489 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:11.508023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.25.123.2:40853)
0928 15:03:11.508076 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:11.508079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:11.508090 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:11.508119 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:11.508122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:11.508138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:11.508191 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:11.508377 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.509063 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.510114 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:11.510507 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:11.510634 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:11.511077 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:11.511112 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:11.511175 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:11.511390 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 15:03:11.511417 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:03:11.512570 1364 raft_consensus.cc:1261] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.512773 1647 raft_consensus.cc:1261] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.512861 1729 consensus_queue.cc:1035] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.513011 1777 consensus_queue.cc:1035] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.515933 1206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:11.516207 1206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:11.517439 1206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Sending DeleteTablet for 3 replicas of tablet ddc276c11294424eb0404fd24617f65a
I20260928 15:03:11.517797 1345 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc276c11294424eb0404fd24617f65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:38328
I20260928 15:03:11.517874 1627 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc276c11294424eb0404fd24617f65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:51260
I20260928 15:03:11.517915 1745 tablet_replica.cc:330] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: stopping tablet replica
I20260928 15:03:11.517966 1746 tablet_replica.cc:330] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: stopping tablet replica
I20260928 15:03:11.517987 1745 raft_consensus.cc:2229] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:11.518013 1746 raft_consensus.cc:2229] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:11.518036 1745 raft_consensus.cc:2258] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.518054 1746 raft_consensus.cc:2258] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.518177 1485 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc276c11294424eb0404fd24617f65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:46592
I20260928 15:03:11.518532 1745 ts_tablet_manager.cc:1905] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.518399 1206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:11.518754 1206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:11.519132 1744 tablet_replica.cc:330] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: stopping tablet replica
I20260928 15:03:11.519265 1744 raft_consensus.cc:2229] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:11.519351 1744 raft_consensus.cc:2258] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.519840 1746 ts_tablet_manager.cc:1905] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.520756 1745 ts_tablet_manager.cc:1918] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:11.520862 1745 log.cc:1196] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/ddc276c11294424eb0404fd24617f65a
I20260928 15:03:11.521106 1745 ts_tablet_manager.cc:1939] T ddc276c11294424eb0404fd24617f65a P c6d66b08cc2f45e58aee35fddaef3811: Deleting consensus metadata
I20260928 15:03:11.521555 1191 catalog_manager.cc:4841] TS c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075): tablet ddc276c11294424eb0404fd24617f65a (table test-table [id=e4d0216f69c640c6b28861c558d54c55]) successfully deleted
I20260928 15:03:11.521883 1485 tablet_service.cc:1467] Processing CreateTablet for tablet f165d4c6753747d6ad807f965d19ec32 (DEFAULT_TABLE table=test-table [id=940c01eda1e34a9c826c891a8dd5d54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:11.521955 1345 tablet_service.cc:1467] Processing CreateTablet for tablet f165d4c6753747d6ad807f965d19ec32 (DEFAULT_TABLE table=test-table [id=940c01eda1e34a9c826c891a8dd5d54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:11.522027 1485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f165d4c6753747d6ad807f965d19ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:11.522042 1345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f165d4c6753747d6ad807f965d19ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:11.522423 1746 ts_tablet_manager.cc:1918] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:11.522509 1746 log.cc:1196] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/ddc276c11294424eb0404fd24617f65a
I20260928 15:03:11.522740 1746 ts_tablet_manager.cc:1939] T ddc276c11294424eb0404fd24617f65a P ae815510be4243cf873bdf2315335f71: Deleting consensus metadata
I20260928 15:03:11.523121 1627 tablet_service.cc:1467] Processing CreateTablet for tablet f165d4c6753747d6ad807f965d19ec32 (DEFAULT_TABLE table=test-table [id=940c01eda1e34a9c826c891a8dd5d54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:11.523269 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f165d4c6753747d6ad807f965d19ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:11.523448 1721 tablet_bootstrap.cc:492] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: Bootstrap starting.
I20260928 15:03:11.523505 1723 tablet_bootstrap.cc:492] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: Bootstrap starting.
I20260928 15:03:11.523882 1723 tablet_bootstrap.cc:654] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:11.523998 1721 tablet_bootstrap.cc:654] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:11.524441 1191 catalog_manager.cc:4841] TS ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837): tablet ddc276c11294424eb0404fd24617f65a (table test-table [id=e4d0216f69c640c6b28861c558d54c55]) successfully deleted
I20260928 15:03:11.524544 1723 tablet_bootstrap.cc:492] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: No bootstrap required, opened a new log
I20260928 15:03:11.524588 1723 ts_tablet_manager.cc:1397] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:11.524742 1723 raft_consensus.cc:348] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.524843 1723 raft_consensus.cc:374] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:11.524873 1723 raft_consensus.cc:729] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Initialized, Role: FOLLOWER
I20260928 15:03:11.524914 1722 tablet_bootstrap.cc:492] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: Bootstrap starting.
I20260928 15:03:11.524922 1723 consensus_queue.cc:260] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.525034 1723 ts_tablet_manager.cc:1428] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 15:03:11.525506 1722 tablet_bootstrap.cc:654] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:11.525542 1721 tablet_bootstrap.cc:492] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: No bootstrap required, opened a new log
I20260928 15:03:11.525612 1721 ts_tablet_manager.cc:1397] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:03:11.525802 1721 raft_consensus.cc:348] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.525887 1721 raft_consensus.cc:374] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:11.525909 1721 raft_consensus.cc:729] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d66b08cc2f45e58aee35fddaef3811, State: Initialized, Role: FOLLOWER
I20260928 15:03:11.525956 1721 consensus_queue.cc:260] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.526051 1721 ts_tablet_manager.cc:1428] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:11.526624 1744 ts_tablet_manager.cc:1905] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.526885 1722 tablet_bootstrap.cc:492] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: No bootstrap required, opened a new log
I20260928 15:03:11.526948 1722 ts_tablet_manager.cc:1397] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:11.527083 1722 raft_consensus.cc:348] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.527156 1722 raft_consensus.cc:374] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:11.527177 1722 raft_consensus.cc:729] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae815510be4243cf873bdf2315335f71, State: Initialized, Role: FOLLOWER
I20260928 15:03:11.527242 1722 consensus_queue.cc:260] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.527344 1722 ts_tablet_manager.cc:1428] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:03:11.528661 1744 ts_tablet_manager.cc:1918] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:11.528721 1744 log.cc:1196] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/ddc276c11294424eb0404fd24617f65a
I20260928 15:03:11.528908 1744 ts_tablet_manager.cc:1939] T ddc276c11294424eb0404fd24617f65a P d4de4781f89143dea24a2b2a541d6737: Deleting consensus metadata
I20260928 15:03:11.529691 1189 catalog_manager.cc:4841] TS d4de4781f89143dea24a2b2a541d6737 (127.25.123.2:40853): tablet ddc276c11294424eb0404fd24617f65a (table test-table [id=e4d0216f69c640c6b28861c558d54c55]) successfully deleted
I20260928 15:03:11.827397 1751 raft_consensus.cc:482] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:11.827527 1751 raft_consensus.cc:504] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.827673 1751 leader_election.cc:288] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075), ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837)
I20260928 15:03:11.827966 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f165d4c6753747d6ad807f965d19ec32" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811" is_pre_election: true
I20260928 15:03:11.828073 1364 raft_consensus.cc:2454] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:11.828029 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f165d4c6753747d6ad807f965d19ec32" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71" is_pre_election: true
I20260928 15:03:11.828159 1647 raft_consensus.cc:2454] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 0.
I20260928 15:03:11.828315 1438 leader_election.cc:302] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6d66b08cc2f45e58aee35fddaef3811, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:11.828454 1751 raft_consensus.cc:2781] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:11.828511 1751 raft_consensus.cc:482] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:11.828544 1751 raft_consensus.cc:3037] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.829427 1751 raft_consensus.cc:504] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.829806 1751 leader_election.cc:288] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Requested vote from peers c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075), ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837)
I20260928 15:03:11.830123 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f165d4c6753747d6ad807f965d19ec32" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae815510be4243cf873bdf2315335f71"
I20260928 15:03:11.830150 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f165d4c6753747d6ad807f965d19ec32" candidate_uuid: "d4de4781f89143dea24a2b2a541d6737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d66b08cc2f45e58aee35fddaef3811"
I20260928 15:03:11.830235 1364 raft_consensus.cc:3037] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.830235 1647 raft_consensus.cc:3037] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:11.831189 1364 raft_consensus.cc:2454] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:11.831212 1647 raft_consensus.cc:2454] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4de4781f89143dea24a2b2a541d6737 in term 1.
I20260928 15:03:11.831453 1438 leader_election.cc:302] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae815510be4243cf873bdf2315335f71, d4de4781f89143dea24a2b2a541d6737; no voters:
I20260928 15:03:11.831597 1751 raft_consensus.cc:2781] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:11.831696 1751 raft_consensus.cc:686] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 1 LEADER]: Becoming Leader. State: Replica: d4de4781f89143dea24a2b2a541d6737, State: Running, Role: LEADER
I20260928 15:03:11.831837 1751 consensus_queue.cc:237] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } }
I20260928 15:03:11.832870 1205 catalog_manager.cc:5495] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4de4781f89143dea24a2b2a541d6737 (127.25.123.2). New cstate: current_term: 1 leader_uuid: "d4de4781f89143dea24a2b2a541d6737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4de4781f89143dea24a2b2a541d6737" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260928 15:03:11.897051 1646 raft_consensus.cc:1261] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.897022 1364 raft_consensus.cc:1261] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Refusing update from remote peer d4de4781f89143dea24a2b2a541d6737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:11.897362 1729 consensus_queue.cc:1035] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae815510be4243cf873bdf2315335f71" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.897495 1777 consensus_queue.cc:1035] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d66b08cc2f45e58aee35fddaef3811" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:11.910164 1206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:11.910300 1206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:11.911263 1206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 501fe93f19c240bab873ad0bb0f640c1: Sending DeleteTablet for 3 replicas of tablet f165d4c6753747d6ad807f965d19ec32
I20260928 15:03:11.911556 1485 tablet_service.cc:1514] Processing DeleteTablet for tablet f165d4c6753747d6ad807f965d19ec32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:46592
I20260928 15:03:11.911597 1345 tablet_service.cc:1514] Processing DeleteTablet for tablet f165d4c6753747d6ad807f965d19ec32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:38328
I20260928 15:03:11.911619 1627 tablet_service.cc:1514] Processing DeleteTablet for tablet f165d4c6753747d6ad807f965d19ec32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:11 UTC) from {username='oryx', principal='oryx/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:51260
I20260928 15:03:11.911696 1744 tablet_replica.cc:330] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: stopping tablet replica
I20260928 15:03:11.911706 1745 tablet_replica.cc:330] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: stopping tablet replica
I20260928 15:03:11.911761 1745 raft_consensus.cc:2229] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:11.911765 1744 raft_consensus.cc:2229] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:11.911810 1745 raft_consensus.cc:2258] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.911808 1746 tablet_replica.cc:330] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: stopping tablet replica
I20260928 15:03:11.911844 1744 raft_consensus.cc:2258] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.911880 1746 raft_consensus.cc:2229] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:11.911935 1746 raft_consensus.cc:2258] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:11.912226 1744 ts_tablet_manager.cc:1905] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.912245 1745 ts_tablet_manager.cc:1905] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.913813 1746 ts_tablet_manager.cc:1905] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:11.914130 1744 ts_tablet_manager.cc:1918] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:11.914193 1744 log.cc:1196] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/f165d4c6753747d6ad807f965d19ec32
I20260928 15:03:11.914171 1745 ts_tablet_manager.cc:1918] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:11.914247 1745 log.cc:1196] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/f165d4c6753747d6ad807f965d19ec32
I20260928 15:03:11.914427 1744 ts_tablet_manager.cc:1939] T f165d4c6753747d6ad807f965d19ec32 P d4de4781f89143dea24a2b2a541d6737: Deleting consensus metadata
I20260928 15:03:11.914575 1745 ts_tablet_manager.cc:1939] T f165d4c6753747d6ad807f965d19ec32 P c6d66b08cc2f45e58aee35fddaef3811: Deleting consensus metadata
I20260928 15:03:11.914817 1189 catalog_manager.cc:4841] TS d4de4781f89143dea24a2b2a541d6737 (127.25.123.2:40853): tablet f165d4c6753747d6ad807f965d19ec32 (table test-table [id=940c01eda1e34a9c826c891a8dd5d54e]) successfully deleted
I20260928 15:03:11.914989 1191 catalog_manager.cc:4841] TS c6d66b08cc2f45e58aee35fddaef3811 (127.25.123.1:40075): tablet f165d4c6753747d6ad807f965d19ec32 (table test-table [id=940c01eda1e34a9c826c891a8dd5d54e]) successfully deleted
I20260928 15:03:11.915493 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 1278
I20260928 15:03:11.917301 1746 ts_tablet_manager.cc:1918] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:03:11.917388 1746 log.cc:1196] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/f165d4c6753747d6ad807f965d19ec32
I20260928 15:03:11.917644 1746 ts_tablet_manager.cc:1939] T f165d4c6753747d6ad807f965d19ec32 P ae815510be4243cf873bdf2315335f71: Deleting consensus metadata
I20260928 15:03:11.918488 1191 catalog_manager.cc:4841] TS ae815510be4243cf873bdf2315335f71 (127.25.123.3:36837): tablet f165d4c6753747d6ad807f965d19ec32 (table test-table [id=940c01eda1e34a9c826c891a8dd5d54e]) successfully deleted
I20260928 15:03:11.923828 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 1420
I20260928 15:03:11.932005 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 1561
I20260928 15:03:11.939724 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 1173
2026-09-28T15:03:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3659 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:12 dist-test-slave-qnz0 krb5kdc[1807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:12 dist-test-slave-qnz0 krb5kdc[1807](info): set up 2 sockets
Sep 28 15:03:12 dist-test-slave-qnz0 krb5kdc[1807](info): commencing operation
krb5kdc: starting...
W20260928 15:03:13.072214 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:03:13 dist-test-slave-qnz0 krb5kdc[1807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:03:13.231982 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:41823
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:13.325392 1820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:13.325687 1820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:13.325730 1820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:13.327099 1820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:13.327198 1820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:13.327222 1820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:13.327243 1820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:13.327271 1820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:13.329010 1820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:41823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:13.329367 1820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:13.329630 1820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:13.335479 1826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:13.335731 1825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:13.335911 1820 server_base.cc:1034] running on GCE node
W20260928 15:03:13.335973 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:13.336238 1820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:13.336603 1820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:13.337822 1820 hybrid_clock.cc:648] HybridClock initialized: now 1790607793337794 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T15:03:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1399 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:13 dist-test-slave-qnz0 krb5kdc[1833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:13 dist-test-slave-qnz0 krb5kdc[1833](info): set up 2 sockets
Sep 28 15:03:13 dist-test-slave-qnz0 krb5kdc[1833](info): commencing operation
krb5kdc: starting...
W20260928 15:03:14.488147 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:03:14 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:03:14.641350 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45537
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:14.732028 1846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:14.732349 1846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:14.732396 1846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:14.733863 1846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:14.733958 1846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:14.733987 1846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:14.734012 1846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:14.734045 1846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:14.735873 1846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:45537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:14.736232 1846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:14.736531 1846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:14.742569 1852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:14.742555 1851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:14.742585 1846 server_base.cc:1034] running on GCE node
W20260928 15:03:14.742555 1854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:14.743079 1846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:14.743443 1846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:14.744611 1846 hybrid_clock.cc:648] HybridClock initialized: now 1790607794744536 us; error 89 us; skew 500 ppm
Sep 28 15:03:14 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:14.747902 1846 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:03:14.748507 1846 webserver.cc:460] Webserver started at http://127.25.123.62:36771/ using document root <none> and password file <none>
I20260928 15:03:14.748793 1846 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:14.748843 1846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:14.748947 1846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:14.750138 1846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "c68b8137896840dab04435a8d26522dd"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-qnz0"
server_key: "7f86186aeb1a1f99cb590f673d245f4a"
server_key_iv: "d698593545b355abfabb4be7c2175bd1"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.750540 1846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "c68b8137896840dab04435a8d26522dd"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-qnz0"
server_key: "7f86186aeb1a1f99cb590f673d245f4a"
server_key_iv: "d698593545b355abfabb4be7c2175bd1"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.752079 1846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:03:14.753139 1862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:14.753489 1846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:14.753716 1846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "c68b8137896840dab04435a8d26522dd"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-qnz0"
server_key: "7f86186aeb1a1f99cb590f673d245f4a"
server_key_iv: "d698593545b355abfabb4be7c2175bd1"
server_key_version: "encryptionkey@0"
I20260928 15:03:14.753854 1846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:14.785089 1846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:14.787753 1846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:14.787961 1846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:14.792798 1846 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:45537
I20260928 15:03:14.792866 1914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:45537 every 8 connection(s)
I20260928 15:03:14.793277 1846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:14.794530 1915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:14.798271 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 1846
I20260928 15:03:14.798417 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:03:14.798652 26092 external_mini_cluster.cc:1428] Setting key 55ac3240c13035b3e173254d170e7560
I20260928 15:03:14.798643 1915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Bootstrap starting.
I20260928 15:03:14.799434 1915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:14.799767 1915 log.cc:824] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:14.800645 1915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: No bootstrap required, opened a new log
I20260928 15:03:14.802987 1915 raft_consensus.cc:348] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:14.803193 1915 raft_consensus.cc:374] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:14.803244 1915 raft_consensus.cc:729] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68b8137896840dab04435a8d26522dd, State: Initialized, Role: FOLLOWER
I20260928 15:03:14.803393 1915 consensus_queue.cc:260] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:14.803481 1915 raft_consensus.cc:388] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:14.803527 1915 raft_consensus.cc:482] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:14.803579 1915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:14.804379 1915 raft_consensus.cc:504] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:14.804558 1915 leader_election.cc:302] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c68b8137896840dab04435a8d26522dd; no voters:
I20260928 15:03:14.804821 1915 leader_election.cc:288] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:14.804850 1920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:14.804973 1920 raft_consensus.cc:686] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 LEADER]: Becoming Leader. State: Replica: c68b8137896840dab04435a8d26522dd, State: Running, Role: LEADER
I20260928 15:03:14.805127 1920 consensus_queue.cc:237] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:14.805130 1915 sys_catalog.cc:564] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 15:03:14 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:14.806784 1921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c68b8137896840dab04435a8d26522dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } } }
I20260928 15:03:14.806998 1921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:14.806854 1922 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c68b8137896840dab04435a8d26522dd. Latest consensus state: current_term: 1 leader_uuid: "c68b8137896840dab04435a8d26522dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } } }
I20260928 15:03:14.807389 1922 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: This master's current role is: LEADER
W20260928 15:03:14.807407 1935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:03:14.807837 1935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:03:14.807922 1937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:14.808341 1937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:14.810115 1937 catalog_manager.cc:1347] Generated new cluster ID: 668386ab65a74e2d92d741ff2b0ee11f
I20260928 15:03:14.810175 1937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:14.811075 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:14.799348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.25.123.62:45537)
0928 15:03:14.799551 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:14.799560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:14.799588 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:03:14.800126 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:14.800135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:14.800189 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:14.800321 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:14.801132 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.802085 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.803257 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:14.803921 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:14.804015 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:14.807100 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:14.807105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:14.807105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:14.807121 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:14.809201 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.809648 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.809651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.809652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.809755 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:14.809975 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:14.809976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:14.809977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:14.810343 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:14.810498 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:14.810694 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 15:03:14.810861 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260928 15:03:14.832360 1937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:14.833570 1937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:14.849232 1937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Generated new TSK 0
I20260928 15:03:14.849534 1937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:03:14.906685 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45537
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:15.019579 1941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:15.019915 1941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:15.019963 1941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:15.021605 1941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:15.021718 1941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:15.021768 1941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:15.024590 1941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:15.025019 1941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:15.025349 1941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:15.032964 1946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.032964 1949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.032968 1947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.034139 1941 server_base.cc:1034] running on GCE node
I20260928 15:03:15.034562 1941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:15.035023 1941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:15.036257 1941 hybrid_clock.cc:648] HybridClock initialized: now 1790607795036214 us; error 60 us; skew 500 ppm
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:15.039341 1941 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:03:15.039978 1941 webserver.cc:460] Webserver started at http://127.25.123.1:34741/ using document root <none> and password file <none>
I20260928 15:03:15.040230 1941 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:15.040282 1941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.040434 1941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:15.041611 1941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "be4923b8c39f43ae85e26afb8c373a6a"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "daf33d62c449ee16a276dc9c11134040"
server_key_iv: "15583cf762b14b87888d4561ed2a24c5"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.042011 1941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "be4923b8c39f43ae85e26afb8c373a6a"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "daf33d62c449ee16a276dc9c11134040"
server_key_iv: "15583cf762b14b87888d4561ed2a24c5"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.043599 1941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:03:15.044914 1957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.045135 1941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:15.045244 1941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "be4923b8c39f43ae85e26afb8c373a6a"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "daf33d62c449ee16a276dc9c11134040"
server_key_iv: "15583cf762b14b87888d4561ed2a24c5"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.045332 1941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.062161 1941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:15.064836 1941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:15.064968 1941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:15.065229 1941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:15.065577 1941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:15.065613 1941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.065635 1941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:15.065649 1941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.073215 1941 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:36223
I20260928 15:03:15.073405 2073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:36223 every 8 connection(s)
I20260928 15:03:15.073760 1941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:15.075667 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 1941
I20260928 15:03:15.075853 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:15.076354 26092 external_mini_cluster.cc:1428] Setting key f0d91748ee63c43c885cf6b63b396a6a
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:15.092555 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.074847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:55229 (local address 127.25.123.62:45537)
0928 15:03:15.075178 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.075181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.075516 (+ 335us) server_negotiation.cc:407] Connection header received
0928 15:03:15.077043 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.077045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.077067 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.077149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.079820 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.080445 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.082607 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.082959 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.083092 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.086950 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.086955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.086955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.086977 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.090098 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.091126 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.091129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.091129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.091296 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.091669 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.091671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.091671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.091828 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.091914 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.092254 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.092446 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260928 15:03:15.092746 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.074696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:55229)
0928 15:03:15.075078 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.075188 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.076779 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.077652 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.077660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.078974 (+ 1314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.079427 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.079439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.080589 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.080592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.082430 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.082436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.083126 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.083127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.083342 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.084123 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.084148 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.086652 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.090319 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.090342 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.090346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.090943 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.091388 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.091390 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.091390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.091582 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.091913 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.091922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.092090 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.092214 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.092446 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:03:15.093554 2074 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45537
I20260928 15:03:15.093688 2074 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:15.093907 2074 heartbeater.cc:511] Master 127.25.123.62:45537 requested a full tablet report, sending...
I20260928 15:03:15.094782 1879 ts_manager.cc:194] Registered new tserver with Master: be4923b8c39f43ae85e26afb8c373a6a (127.25.123.1:36223)
I20260928 15:03:15.095630 1879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:55229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:03:15.126832 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45537
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:15.228942 2079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:15.229259 2079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:15.229307 2079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:15.230698 2079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:15.230799 2079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:15.230849 2079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:15.232568 2079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:15.232910 2079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:15.233183 2079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:15.239223 2085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.239224 2084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.239617 2079 server_base.cc:1034] running on GCE node
W20260928 15:03:15.239449 2087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.240307 2079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:15.240890 2079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:15.242183 2079 hybrid_clock.cc:648] HybridClock initialized: now 1790607795242230 us; error 130 us; skew 500 ppm
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:15.245927 2079 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:03:15.246542 2079 webserver.cc:460] Webserver started at http://127.25.123.2:42765/ using document root <none> and password file <none>
I20260928 15:03:15.246826 2079 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:15.246881 2079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.247031 2079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:15.248472 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "1fab0654e4154a5b9dd8782a93dc4888"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "c41248a5a86e526e11d3068d2c9e4de4"
server_key_iv: "0f94db8773f68867d2bd5b66588d4072"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.248915 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "1fab0654e4154a5b9dd8782a93dc4888"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "c41248a5a86e526e11d3068d2c9e4de4"
server_key_iv: "0f94db8773f68867d2bd5b66588d4072"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.250586 2079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:03:15.251583 2094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.251999 2079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:15.252089 2079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "1fab0654e4154a5b9dd8782a93dc4888"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "c41248a5a86e526e11d3068d2c9e4de4"
server_key_iv: "0f94db8773f68867d2bd5b66588d4072"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.252223 2079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.268359 2079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:15.272331 2079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:15.272746 2079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:15.273110 2079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:15.273594 2079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:15.273638 2079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.273669 2079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:15.273682 2079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.282562 2079 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:35395
I20260928 15:03:15.282685 2209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:35395 every 8 connection(s)
I20260928 15:03:15.283073 2079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:15.284324 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 2079
I20260928 15:03:15.284498 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:15.284704 26092 external_mini_cluster.cc:1428] Setting key ee38628f824478443bf92ca706b467ce
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:03:15.297372 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.284007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:45931)
0928 15:03:15.284172 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.284220 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.284945 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.285266 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.285271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.286088 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.286322 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.286327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.287493 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.287494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.288818 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.288822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.289412 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.289413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.289558 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.290189 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.290199 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.291884 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.294532 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.294544 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.294546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.294910 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.295511 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.295512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.295512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.295650 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.296563 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.296572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.296785 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.296913 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.297088 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260928 15:03:15.297441 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.284092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:45931 (local address 127.25.123.62:45537)
0928 15:03:15.284160 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.284163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.284330 (+ 167us) server_negotiation.cc:407] Connection header received
0928 15:03:15.285071 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.285072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.285085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.285166 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.286462 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.287363 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.288934 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.289286 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.289377 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.292064 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.292066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.292066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.292076 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.294385 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.295294 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.295296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.295296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.295375 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.295898 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.295901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.295901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.296385 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.296536 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.297185 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.297341 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 15:03:15.299386 2210 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
I20260928 15:03:15.299676 2210 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:15.300249 2210 heartbeater.cc:511] Master 127.25.123.62:45537 requested a full tablet report, sending...
I20260928 15:03:15.301154 1879 ts_manager.cc:194] Registered new tserver with Master: 1fab0654e4154a5b9dd8782a93dc4888 (127.25.123.2:35395)
I20260928 15:03:15.302009 1879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:45931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:03:15.320994 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45537
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:15.425436 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:15.425803 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:15.425854 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:15.428956 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:15.429044 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:15.429100 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:15.432806 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:15.433214 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:15.433565 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:15.440106 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.440162 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.440269 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.440475 2215 server_base.cc:1034] running on GCE node
I20260928 15:03:15.440780 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:15.441080 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:15.442301 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790607795442274 us; error 49 us; skew 500 ppm
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:15.445428 2215 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:03:15.446156 2215 webserver.cc:460] Webserver started at http://127.25.123.3:37809/ using document root <none> and password file <none>
I20260928 15:03:15.446422 2215 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:15.446499 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.446662 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:15.448022 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "2d97fb402c3a4e90a43674ab6b41ccb0"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "d8a0941098934f0c47f704d85f1987c0"
server_key_iv: "9317cda48eaf147475159e5caa713534"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.448607 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "2d97fb402c3a4e90a43674ab6b41ccb0"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "d8a0941098934f0c47f704d85f1987c0"
server_key_iv: "9317cda48eaf147475159e5caa713534"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.450551 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:03:15.452289 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.452728 2215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:15.452873 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "2d97fb402c3a4e90a43674ab6b41ccb0"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "d8a0941098934f0c47f704d85f1987c0"
server_key_iv: "9317cda48eaf147475159e5caa713534"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.453162 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.474234 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:15.477548 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:15.477691 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:15.478016 2215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:15.478498 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:15.478549 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.478611 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:15.478638 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.485821 2215 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:46847
I20260928 15:03:15.485865 2346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:46847 every 8 connection(s)
I20260928 15:03:15.486250 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:15.488956 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 2215
I20260928 15:03:15.489086 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:15.489279 26092 external_mini_cluster.cc:1428] Setting key f28abe3ab2b965266ddd2ef27533adea
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:15.498183 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.487102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:43427 (local address 127.25.123.62:45537)
0928 15:03:15.487183 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:15.487185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:15.487408 (+ 223us) server_negotiation.cc:407] Connection header received
0928 15:03:15.487972 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:15.487973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:15.487985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:15.488053 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:15.489621 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.490224 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.491477 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.491730 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.491840 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.494935 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:15.494938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:15.494938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:15.494947 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:15.496587 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.497088 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.497090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.497090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.497156 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:15.497429 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:15.497430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:15.497431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:15.497564 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:15.497650 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:15.497982 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 15:03:15.498075 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:03:15.498431 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:15.487065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:43427)
0928 15:03:15.487241 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:15.487285 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:15.487847 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:15.488242 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:15.488247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:15.489129 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:15.489453 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.489460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.490370 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.490371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.491343 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:15.491346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:15.491856 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:15.491857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:15.492011 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:15.493127 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:15.493137 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:15.494719 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:15.496719 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.496731 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.496734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.497003 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.497233 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:15.497234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:15.497234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:15.497360 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:15.497640 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:15.497646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:15.497810 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 15:03:15.497957 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 15:03:15.498150 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260928 15:03:15.499372 2347 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45537
I20260928 15:03:15.499488 2347 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:15.499657 2347 heartbeater.cc:511] Master 127.25.123.62:45537 requested a full tablet report, sending...
I20260928 15:03:15.500605 1879 ts_manager.cc:194] Registered new tserver with Master: 2d97fb402c3a4e90a43674ab6b41ccb0 (127.25.123.3:46847)
I20260928 15:03:15.501153 1879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:43427
I20260928 15:03:15.501526 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab.
Entry for principal oryx/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab.
WARNING: no policy specified for oryx/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab.
Entry for principal oryx/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab.
WARNING: no policy specified for oryx/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab.
Entry for principal oryx/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab.
I20260928 15:03:15.609988 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 1941
I20260928 15:03:15.617206 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 2079
I20260928 15:03:15.622143 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 2215
I20260928 15:03:15.627205 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 1846
I20260928 15:03:15.633714 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45537
--webserver_interface=127.25.123.62
--webserver_port=36771
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:15.738541 2356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:15.738797 2356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:15.738830 2356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:15.740155 2356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:15.740223 2356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:15.740240 2356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:15.740253 2356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:15.740266 2356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:15.742527 2356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:45537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=36771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:15.742884 2356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:15.743202 2356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:15.749809 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.749846 2364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.749854 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.750445 2356 server_base.cc:1034] running on GCE node
I20260928 15:03:15.750893 2356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:15.751318 2356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:15.752502 2356 hybrid_clock.cc:648] HybridClock initialized: now 1790607795752487 us; error 140 us; skew 500 ppm
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:15.756332 2356 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:03:15.757282 2356 webserver.cc:460] Webserver started at http://127.25.123.62:36771/ using document root <none> and password file <none>
I20260928 15:03:15.757647 2356 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:15.757725 2356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:15.759582 2356 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 15:03:15.760753 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:15.761140 2356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:15.761405 2356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "c68b8137896840dab04435a8d26522dd"
format_stamp: "Formatted at 2026-09-28 15:03:14 on dist-test-slave-qnz0"
server_key: "7f86186aeb1a1f99cb590f673d245f4a"
server_key_iv: "d698593545b355abfabb4be7c2175bd1"
server_key_version: "encryptionkey@0"
I20260928 15:03:15.761974 2356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:15.785411 2356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:15.788144 2356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:15.788347 2356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:15.796761 2356 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:45537
I20260928 15:03:15.796933 2425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:45537 every 8 connection(s)
I20260928 15:03:15.797566 2356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:15.799649 2426 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:03:15.801429 2426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Bootstrap starting.
I20260928 15:03:15.802372 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 2356
I20260928 15:03:15.802757 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:36223
--local_ip_for_outbound_sockets=127.25.123.1
--tserver_master_addrs=127.25.123.62:45537
--webserver_port=34741
--webserver_interface=127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:03:15.816350 2426 log.cc:824] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:15.819391 2426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:03:15.822198 2426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Bootstrap complete.
I20260928 15:03:15.825665 2426 raft_consensus.cc:348] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:15.825979 2426 raft_consensus.cc:729] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68b8137896840dab04435a8d26522dd, State: Initialized, Role: FOLLOWER
I20260928 15:03:15.826210 2426 consensus_queue.cc:260] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:15.826319 2426 raft_consensus.cc:388] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:15.826378 2426 raft_consensus.cc:482] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:15.826429 2426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:03:15.827598 2426 raft_consensus.cc:504] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:15.827742 2426 leader_election.cc:302] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c68b8137896840dab04435a8d26522dd; no voters:
I20260928 15:03:15.828001 2426 leader_election.cc:288] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:03:15.828044 2430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:03:15.828181 2430 raft_consensus.cc:686] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [term 2 LEADER]: Becoming Leader. State: Replica: c68b8137896840dab04435a8d26522dd, State: Running, Role: LEADER
I20260928 15:03:15.828305 2430 consensus_queue.cc:237] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } }
I20260928 15:03:15.828313 2426 sys_catalog.cc:564] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:15.828733 2431 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c68b8137896840dab04435a8d26522dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } } }
I20260928 15:03:15.828812 2431 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:15.828976 2432 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c68b8137896840dab04435a8d26522dd. Latest consensus state: current_term: 2 leader_uuid: "c68b8137896840dab04435a8d26522dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b8137896840dab04435a8d26522dd" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45537 } } }
I20260928 15:03:15.829036 2432 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:15.829681 2434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:15.830369 2434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:15.830528 2434 catalog_manager.cc:1259] Loaded cluster ID: 668386ab65a74e2d92d741ff2b0ee11f
I20260928 15:03:15.830574 2434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:15.833082 2434 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:15.833917 2434 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c68b8137896840dab04435a8d26522dd: Loaded TSK: 0
I20260928 15:03:15.834568 2434 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:03:15.950608 2428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:15.951122 2428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:15.951159 2428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:15.953490 2428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:15.953934 2428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:15.954039 2428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:15.964486 2428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.1.keytab
--principal=oryx/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:36223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=34741
--tserver_master_addrs=127.25.123.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:15.964895 2428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:15.965510 2428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:15.981499 2455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.981974 2452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:15.982208 2453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:15.991202 2428 server_base.cc:1034] running on GCE node
I20260928 15:03:15.991539 2428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:15.991901 2428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:15.993161 2428 hybrid_clock.cc:648] HybridClock initialized: now 1790607795993122 us; error 53 us; skew 500 ppm
Sep 28 15:03:15 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607795, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:15.999763 2428 init.cc:377] Logged in from keytab as oryx/127.25.123.1@KRBTEST.COM (short username oryx)
I20260928 15:03:16.000623 2428 webserver.cc:460] Webserver started at http://127.25.123.1:34741/ using document root <none> and password file <none>
I20260928 15:03:16.001000 2428 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:16.001094 2428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:16.005628 2428 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.003s
I20260928 15:03:16.006937 2462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.007186 2428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:16.007289 2428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "be4923b8c39f43ae85e26afb8c373a6a"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "daf33d62c449ee16a276dc9c11134040"
server_key_iv: "15583cf762b14b87888d4561ed2a24c5"
server_key_version: "encryptionkey@0"
I20260928 15:03:16.007750 2428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:16.046391 2428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:16.049221 2428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:16.049647 2428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:16.051363 2428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:16.051887 2428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:16.052038 2428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.052115 2428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:16.052142 2428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.067124 2428 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:36223
I20260928 15:03:16.067342 2577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:36223 every 8 connection(s)
I20260928 15:03:16.067701 2428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:16.074044 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 2428
I20260928 15:03:16.074609 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:35395
--local_ip_for_outbound_sockets=127.25.123.2
--tserver_master_addrs=127.25.123.62:45537
--webserver_port=42765
--webserver_interface=127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:16.085198 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.068766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:34683)
0928 15:03:16.069075 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.069146 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.070053 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.071113 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.071118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.071800 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.072035 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.072041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.074977 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.074978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.077094 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.077101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.078782 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.078790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.079049 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.079954 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.079967 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.084852 (+ 4885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
W20260928 15:03:16.086901 2578 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:45537 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.123.62:45537: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
W20260928 15:03:16.087308 2581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.068792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:34683 (local address 127.25.123.62:45537)
0928 15:03:16.070362 (+ 1570us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.070373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.070420 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:03:16.070572 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.070578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.070642 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.071019 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.072913 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.074786 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.077354 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.078496 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.078878 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.085160 (+ 6282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:34683: BlockingRecv error: recv got EOF from 127.25.123.1:34683 (error 108)
Metrics: {"server-negotiator.queue_time_us":1507,"thread_start_us":98,"threads_started":1}
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:16.098920 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.087333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:37047)
0928 15:03:16.087472 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.087488 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.087545 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.089332 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.089334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.089565 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.089665 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.089668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.091511 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.091513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.092933 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.092939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.093783 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.093784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.093885 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.094833 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.094840 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.098812 (+ 3972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260928 15:03:16.100128 2583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.087501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:37047 (local address 127.25.123.62:45537)
0928 15:03:16.087931 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.087934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.087951 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:03:16.088089 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.088094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.088152 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.088294 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.090276 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.091374 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.093169 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.093467 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.093789 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.099534 (+ 5745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:37047: BlockingRecv error: recv got EOF from 127.25.123.1:37047 (error 108)
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":325,"threads_started":1}
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:16.108798 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.100179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:50429)
0928 15:03:16.101314 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.101333 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.101465 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.101868 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.101870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.102040 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.102087 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.102088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.103460 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.103462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.103987 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.103989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.104442 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.104443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.104516 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.105055 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.105061 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.108693 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":977}
W20260928 15:03:16.108974 2581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.100936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:50429 (local address 127.25.123.62:45537)
0928 15:03:16.101008 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.101012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.101428 (+ 416us) server_negotiation.cc:407] Connection header received
0928 15:03:16.101567 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.101570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.101600 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.101675 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.102178 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.103286 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.104097 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.104309 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.104439 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.108820 (+ 4381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:50429: BlockingRecv error: recv got EOF from 127.25.123.1:50429 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 15:03:16.108978 2578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:03:16.225790 2582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:16.226126 2582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:16.226181 2582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:16.228237 2582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:16.228657 2582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:16.228801 2582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:16.234162 2582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.2.keytab
--principal=oryx/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:35395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=42765
--tserver_master_addrs=127.25.123.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:16.234606 2582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:16.235651 2582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:16.245103 2589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.245147 2588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.245103 2591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:16.247856 2582 server_base.cc:1034] running on GCE node
I20260928 15:03:16.248172 2582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:16.248556 2582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:16.249838 2582 hybrid_clock.cc:648] HybridClock initialized: now 1790607796249783 us; error 77 us; skew 500 ppm
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:16.253413 2582 init.cc:377] Logged in from keytab as oryx/127.25.123.2@KRBTEST.COM (short username oryx)
I20260928 15:03:16.254393 2582 webserver.cc:460] Webserver started at http://127.25.123.2:42765/ using document root <none> and password file <none>
I20260928 15:03:16.254770 2582 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:16.254843 2582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:16.258553 2582 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:16.260111 2598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.260735 2582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:16.260900 2582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "1fab0654e4154a5b9dd8782a93dc4888"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "c41248a5a86e526e11d3068d2c9e4de4"
server_key_iv: "0f94db8773f68867d2bd5b66588d4072"
server_key_version: "encryptionkey@0"
I20260928 15:03:16.261482 2582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:16.281042 2582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:16.284782 2582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:16.284979 2582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:16.285259 2582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:16.285646 2582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:16.285708 2582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.285755 2582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:16.285784 2582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.301350 2582 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:35395
I20260928 15:03:16.301388 2712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:35395 every 8 connection(s)
I20260928 15:03:16.301856 2582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:16.307685 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 2582
I20260928 15:03:16.308043 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:46847
--local_ip_for_outbound_sockets=127.25.123.3
--tserver_master_addrs=127.25.123.62:45537
--webserver_port=37809
--webserver_interface=127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:16.315510 2581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.303045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:50371 (local address 127.25.123.62:45537)
0928 15:03:16.303114 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.303116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.304024 (+ 908us) server_negotiation.cc:407] Connection header received
0928 15:03:16.306314 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.306315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.306336 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.306417 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.307549 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.308417 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.310408 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.310646 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.310785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.315441 (+ 4656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:50371: BlockingRecv error: recv got EOF from 127.25.123.2:50371 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 15:03:16.315594 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.302837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:50371)
0928 15:03:16.303063 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.303762 (+ 699us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.306030 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.306634 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.306639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.307078 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.307335 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.307341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.308552 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.308554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.310072 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.310077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.310805 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.310806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.311175 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.312111 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.312125 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.315332 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
W20260928 15:03:16.315910 2713 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:45537 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.123.62:45537: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:16.326313 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.316205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:39531)
0928 15:03:16.316266 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.316278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.316328 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.317042 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.317043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.317205 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.317265 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.317267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.318861 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.318862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.319784 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.319788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.320728 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.320730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.320814 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.322360 (+ 1546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.322368 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.326222 (+ 3854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 15:03:16.326704 2581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.316690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:39531 (local address 127.25.123.62:45537)
0928 15:03:16.316833 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.316835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.316845 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:16.316884 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.316885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.316898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.316983 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.317423 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.318610 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.319952 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.320557 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.320714 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.326637 (+ 5923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:39531: BlockingRecv error: recv got EOF from 127.25.123.2:39531 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:16.335227 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.326684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:34439)
0928 15:03:16.326882 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.326892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.326948 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.327110 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.327111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.327279 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.327333 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.327335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.328357 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.328359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.329448 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.329451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.330169 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.330170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.330238 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.331625 (+ 1387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.331632 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.335141 (+ 3509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180}
W20260928 15:03:16.335438 2583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.326787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:34439 (local address 127.25.123.62:45537)
0928 15:03:16.326849 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.326852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.326958 (+ 106us) server_negotiation.cc:407] Connection header received
0928 15:03:16.327007 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.327007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.327020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.327078 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.327495 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.328210 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.329551 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.330065 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.330154 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.335353 (+ 5199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:34439: BlockingRecv error: recv got EOF from 127.25.123.2:34439 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 15:03:16.335578 2713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:03:16.434398 2716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:16.434692 2716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:16.434839 2716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:16.437232 2716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:16.437400 2716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:16.437667 2716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:16.441874 2716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/krb5kdc/oryx_127.25.123.3.keytab
--principal=oryx/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:46847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=37809
--tserver_master_addrs=127.25.123.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:16.442307 2716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:16.442587 2716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:16.450302 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.450440 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:16.450507 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:16.450889 2716 server_base.cc:1034] running on GCE node
I20260928 15:03:16.451464 2716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:16.451910 2716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:16.453163 2716 hybrid_clock.cc:648] HybridClock initialized: now 1790607796453036 us; error 149 us; skew 500 ppm
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607796, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:16.459043 2716 init.cc:377] Logged in from keytab as oryx/127.25.123.3@KRBTEST.COM (short username oryx)
I20260928 15:03:16.460055 2716 webserver.cc:460] Webserver started at http://127.25.123.3:37809/ using document root <none> and password file <none>
I20260928 15:03:16.460978 2716 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:16.461136 2716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:16.463889 2716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:03:16.465530 2731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.465889 2716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:16.466010 2716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "2d97fb402c3a4e90a43674ab6b41ccb0"
format_stamp: "Formatted at 2026-09-28 15:03:15 on dist-test-slave-qnz0"
server_key: "d8a0941098934f0c47f704d85f1987c0"
server_key_iv: "9317cda48eaf147475159e5caa713534"
server_key_version: "encryptionkey@0"
I20260928 15:03:16.466594 2716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:16.484378 2716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:16.488615 2716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:16.488867 2716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:16.489156 2716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:16.489778 2716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:16.489854 2716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.489892 2716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:16.489925 2716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:16.503331 2716 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:46847
I20260928 15:03:16.503433 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:46847 every 8 connection(s)
I20260928 15:03:16.503907 2716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:16.507496 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 2716
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:03:16.521679 2583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.505406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:48371 (local address 127.25.123.62:45537)
0928 15:03:16.505499 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.505501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.506344 (+ 843us) server_negotiation.cc:407] Connection header received
0928 15:03:16.507952 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.507954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.507978 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.508054 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.509924 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.510521 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.512565 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.513046 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.513146 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.521044 (+ 7898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:48371: BlockingRecv error: recv got EOF from 127.25.123.3:48371 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:03:16.521836 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.505413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:48371)
0928 15:03:16.506022 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.506065 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.507552 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.508242 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.508263 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.509091 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.509556 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.509564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.510712 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.510714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.512363 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.512370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.513417 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.513419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.513593 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.514524 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.514540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.520909 (+ 6369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":337,"threads_started":1}
W20260928 15:03:16.522130 2848 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:45537 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.123.62:45537: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
I20260928 15:03:16.523368 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.508480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.25.123.62:45537)
0928 15:03:16.508725 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.508729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.508755 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:03:16.508798 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.508800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.508817 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.508964 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.509723 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.510625 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.512322 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.512905 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.512978 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.515910 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:16.515917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:16.515917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:16.515937 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:16.521306 (+ 5369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.522009 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.522013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.522014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.522168 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:16.522429 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:16.522431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:16.522431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:16.522798 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:16.522989 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:16.523107 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 15:03:16.523249 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:16.533279 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.522525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:35571)
0928 15:03:16.522590 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.522606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.522691 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.522872 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.522873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.523070 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.523156 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.523158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.524452 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.524455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.525174 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.525178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.525773 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.525774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.525864 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.527003 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.527010 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.533181 (+ 6171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 15:03:16.533471 2583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.522605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:35571 (local address 127.25.123.62:45537)
0928 15:03:16.522692 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.522695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.522706 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:16.522761 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.522762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.522780 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.522873 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.523249 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.524289 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.525373 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.525682 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.525774 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.533386 (+ 7612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:35571: BlockingRecv error: recv got EOF from 127.25.123.3:35571 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:16 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:16.543995 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:16.534455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:37651)
0928 15:03:16.534544 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:16.534560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:16.534621 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:16.535361 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:16.535363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:16.535565 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:16.535613 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.535615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.538074 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.538077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.538881 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:16.538885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.539347 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.539348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:16.539403 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.540146 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:16.540152 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:16.543932 (+ 3780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 15:03:16.544639 2848 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:03:16.544894 2583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:16.534562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:37651 (local address 127.25.123.62:45537)
0928 15:03:16.534664 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:16.534667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:16.534679 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:16.534735 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:16.534736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:16.534749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:16.535350 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:16.536334 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.537951 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.538991 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:16.539255 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:16.539349 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:16.544832 (+ 5483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:37651: BlockingRecv error: recv got EOF from 127.25.123.3:37651 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 15:03:17 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:17 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:17.117638 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.109795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:59821)
0928 15:03:17.110181 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.110197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.110302 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.110636 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.110638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.110873 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:17.110945 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.110948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.112059 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.112061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.112774 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.112777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.113579 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.113581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.113638 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.114618 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:17.114623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:17.117528 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":127,"threads_started":1}
W20260928 15:03:17.118234 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:17.109987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:59821 (local address 127.25.123.62:45537)
0928 15:03:17.110114 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.110117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.110335 (+ 218us) server_negotiation.cc:407] Connection header received
0928 15:03:17.110381 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.110382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.110399 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.110497 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.111205 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.111775 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.112959 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.113285 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.113518 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.118129 (+ 4611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:59821: BlockingRecv error: recv got EOF from 127.25.123.1:59821 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
Sep 28 15:03:17 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:17 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:17.343446 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.336177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:51141)
0928 15:03:17.336407 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.336420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.336483 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.336947 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.336949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.337286 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:17.337354 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.337357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.338234 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.338235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.339007 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.339010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.339806 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.339809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.339874 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.340735 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:17.340742 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:17.343368 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":131,"threads_started":1}
W20260928 15:03:17.343745 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:17.336353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:51141 (local address 127.25.123.62:45537)
0928 15:03:17.336430 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.336433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.336477 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:03:17.336651 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.336654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.336679 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.336788 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.337512 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.338119 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.339211 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.339645 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.339793 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.343575 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:51141: BlockingRecv error: recv got EOF from 127.25.123.2:51141 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 15:03:17 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:17 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:17.551813 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:17.545082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:41637)
0928 15:03:17.545248 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:17.545260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:17.545335 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:17.545544 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:17.545545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:17.545790 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:17.545864 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.546015 (+ 151us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.546954 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.546955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.547508 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:17.547510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.548471 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.548475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:17.548535 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.549485 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:17.549490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:17.551737 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
W20260928 15:03:17.552050 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:17.545154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:41637 (local address 127.25.123.62:45537)
0928 15:03:17.545208 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:17.545210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:17.545330 (+ 120us) server_negotiation.cc:407] Connection header received
0928 15:03:17.545412 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:17.545413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:17.545430 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:17.545493 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:17.546136 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.546831 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.547757 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:17.548240 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:17.548328 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:17.551968 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:41637: BlockingRecv error: recv got EOF from 127.25.123.3:41637 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 15:03:18 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:18 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:18.126315 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.118287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:43265)
0928 15:03:18.118458 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:18.118488 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:18.118563 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:18.118742 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:18.118743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:18.118960 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:18.119010 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:18.119012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.120154 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.120156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:18.121181 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:18.121188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.122011 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.122013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:18.122106 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.123408 (+ 1302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:18.123414 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:18.126242 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
W20260928 15:03:18.126816 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:18.118382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:43265 (local address 127.25.123.62:45537)
0928 15:03:18.118497 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:18.118499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:18.118546 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:03:18.118621 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:18.118622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:18.118634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:18.118739 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:18.119212 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.119916 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.121565 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.121817 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.121916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.126729 (+ 4813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:43265: BlockingRecv error: recv got EOF from 127.25.123.1:43265 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
Sep 28 15:03:18 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:18 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:18.351094 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.344321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:51827)
0928 15:03:18.344678 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:18.344709 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:18.344810 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:18.345151 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:18.345153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:18.345580 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:18.345650 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:18.345654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.346458 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.346459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:18.347071 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:18.347073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.347565 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.347566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:18.347616 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.348612 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:18.348619 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:18.351026 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":188,"threads_started":1}
W20260928 15:03:18.351289 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:18.344446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:51827 (local address 127.25.123.62:45537)
0928 15:03:18.344551 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:18.344554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:18.344881 (+ 327us) server_negotiation.cc:407] Connection header received
0928 15:03:18.344924 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:18.344925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:18.344944 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:18.345021 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:18.345816 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.346364 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.347180 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.347383 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.347453 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.351220 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:51827: BlockingRecv error: recv got EOF from 127.25.123.2:51827 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 15:03:18 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:18 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:18.569051 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:18.552545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:56219)
0928 15:03:18.552690 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:18.552704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:18.552783 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:18.553069 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:18.553070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:18.553237 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:18.553301 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:18.553302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.554498 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.554499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:18.555205 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:18.555208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.555924 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.555926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:18.556625 (+ 699us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.564628 (+ 8003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:18.564641 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:18.568961 (+ 4320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
W20260928 15:03:18.569406 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:18.552746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:56219 (local address 127.25.123.62:45537)
0928 15:03:18.552903 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:18.552906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:18.552924 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:03:18.552952 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:18.552953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:18.552965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:18.553023 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:18.553472 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.554299 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.555385 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:18.555631 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:18.555731 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:18.569341 (+ 13610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:56219: BlockingRecv error: recv got EOF from 127.25.123.3:56219 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 28 15:03:19 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:19 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:19.134232 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.127211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:42021)
0928 15:03:19.127349 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.127365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:19.127444 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:19.127623 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:19.127624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:19.127819 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:19.127870 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.127872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.129054 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.129058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.130216 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.130222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.130767 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.130768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.130837 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.131680 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:19.131686 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:19.134090 (+ 2404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
W20260928 15:03:19.134292 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:19.127318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:42021 (local address 127.25.123.62:45537)
0928 15:03:19.127468 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:19.127470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:19.127480 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:19.127508 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:19.127509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:19.127521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:19.127615 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:19.127966 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.128715 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.130350 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.130676 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.130767 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.134205 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:42021: BlockingRecv error: recv got EOF from 127.25.123.1:42021 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
Sep 28 15:03:19 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:19 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:19.358475 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.351744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:38733)
0928 15:03:19.352054 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.352067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:19.352198 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:19.352482 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:19.352483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:19.352673 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:19.352743 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.352745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.353566 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.353567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.354142 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.354143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.354526 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.354527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.354580 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.355456 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:19.355465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:19.358336 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":214,"threads_started":1}
W20260928 15:03:19.358628 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:19.351909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:38733 (local address 127.25.123.62:45537)
0928 15:03:19.352050 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:19.352052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:19.352174 (+ 122us) server_negotiation.cc:407] Connection header received
0928 15:03:19.352335 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:19.352336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:19.352351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:19.352467 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:19.352890 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.353439 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.354240 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.354444 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.354528 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.358526 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:38733: BlockingRecv error: recv got EOF from 127.25.123.2:38733 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 15:03:19 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:19 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:19.576359 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:19.569923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:60341)
0928 15:03:19.570086 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:19.570097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:19.570148 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:19.570344 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:19.570344 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:19.570510 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:19.570570 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.570576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.571474 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.571476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.572098 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:19.572100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.572651 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.572652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:19.572700 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.573604 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:19.573610 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:19.576293 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
W20260928 15:03:19.577088 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:19.570039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:60341 (local address 127.25.123.62:45537)
0928 15:03:19.570120 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:19.570122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:19.570138 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:03:19.570204 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:19.570205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:19.570217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:19.570286 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:19.570802 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.571370 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.572255 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:19.572546 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:19.572616 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:19.577022 (+ 4406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:60341: BlockingRecv error: recv got EOF from 127.25.123.3:60341 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 15:03:20 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:20 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:20.141388 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:20.135056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:34825)
0928 15:03:20.135200 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:20.135220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:20.135303 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:20.135512 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:20.135513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:20.135687 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:20.135740 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:20.135742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.136647 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.136649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:20.137255 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:20.137257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.137975 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.137976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:20.138035 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:20.138625 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:20.138630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:20.141320 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260928 15:03:20.141968 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:20.135227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:34825 (local address 127.25.123.62:45537)
0928 15:03:20.135353 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:20.135355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:20.135365 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:20.135400 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:20.135401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:20.135411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:20.135481 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:20.135871 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.136530 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.137510 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.137823 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.137942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:20.141877 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:34825: BlockingRecv error: recv got EOF from 127.25.123.1:34825 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
Sep 28 15:03:20 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:20 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:20.365530 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:20.359301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:58165)
0928 15:03:20.359618 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:20.359638 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:20.359763 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:20.360035 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:20.360037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:20.360292 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:20.360362 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:20.360365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.361490 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.361492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:20.362193 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:20.362196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.362753 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.362754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:20.362825 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:20.363478 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:20.363483 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:20.365463 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":143,"threads_started":1}
W20260928 15:03:20.365880 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:20.359465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:58165 (local address 127.25.123.62:45537)
0928 15:03:20.359594 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:20.359597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:20.359850 (+ 253us) server_negotiation.cc:407] Connection header received
0928 15:03:20.359895 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:20.359897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:20.359915 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:20.359987 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:20.360501 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.361130 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.362457 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.362654 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.362722 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:20.365799 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:58165: BlockingRecv error: recv got EOF from 127.25.123.2:58165 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 15:03:20 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:20 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:20.585469 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:20.577840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:50595)
0928 15:03:20.577974 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:20.577995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:20.578063 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:20.578306 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:20.578308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:20.578477 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:20.578526 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:20.578528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.579379 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.579380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:20.580234 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:20.580246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.581362 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.581363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:20.581423 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:20.582289 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:20.582294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:20.585400 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
W20260928 15:03:20.585944 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:20.577876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:50595 (local address 127.25.123.62:45537)
0928 15:03:20.578004 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:20.578006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:20.578078 (+ 72us) server_negotiation.cc:407] Connection header received
0928 15:03:20.578154 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:20.578155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:20.578170 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:20.578240 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:20.578645 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.579213 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.580854 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:20.581097 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:20.581168 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:20.585870 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:50595: BlockingRecv error: recv got EOF from 127.25.123.3:50595 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 15:03:21 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:21 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:21.148173 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.141954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:46605)
0928 15:03:21.142127 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.142139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:21.142194 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:21.142395 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:21.142397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:21.142602 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:21.142722 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.142725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.143522 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.143524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.144076 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.144078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.144625 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.144627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.144713 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.145383 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:21.145388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:21.148083 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
W20260928 15:03:21.148434 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:21.142003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:46605 (local address 127.25.123.62:45537)
0928 15:03:21.142208 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:21.142210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:21.142218 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:03:21.142242 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:21.142243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:21.142256 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:21.142335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:21.142859 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.143422 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.144189 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.144424 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.144491 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.148241 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:46605: BlockingRecv error: recv got EOF from 127.25.123.1:46605 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
Sep 28 15:03:21 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:21 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:21.372599 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.366290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:59777)
0928 15:03:21.366502 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.366513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:21.366580 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:21.366844 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:21.366847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:21.367099 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:21.367178 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.367180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.368059 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.368060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.368725 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.368728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.369408 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.369410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.369506 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.370291 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:21.370297 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:21.372531 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
W20260928 15:03:21.372750 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:21.366467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:59777 (local address 127.25.123.62:45537)
0928 15:03:21.366519 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:21.366526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:21.366631 (+ 105us) server_negotiation.cc:407] Connection header received
0928 15:03:21.366654 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:21.366655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:21.366667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:21.366731 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:21.367333 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.367932 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.368912 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.369165 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.369321 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.372669 (+ 3348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:59777: BlockingRecv error: recv got EOF from 127.25.123.2:59777 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 15:03:21 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:21 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:21.593595 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:21.586526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:49235)
0928 15:03:21.586749 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:21.586768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:21.586844 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:21.587176 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:21.587177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:21.587939 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:21.588004 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.588006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.589059 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.589060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.589830 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:21.589834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.590515 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.590516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:21.590589 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.591380 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:21.591386 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:21.593529 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":153,"threads_started":1}
W20260928 15:03:21.593875 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:21.586646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:49235 (local address 127.25.123.62:45537)
0928 15:03:21.586722 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:21.586724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:21.586914 (+ 190us) server_negotiation.cc:407] Connection header received
0928 15:03:21.586971 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:21.586972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:21.586996 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:21.587105 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:21.588277 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.588890 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.590009 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:21.590375 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:21.590469 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:21.593812 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:49235: BlockingRecv error: recv got EOF from 127.25.123.3:49235 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 15:03:22 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:22 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:22.155890 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.149061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:56541)
0928 15:03:22.149261 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.149273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:22.149336 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:22.149610 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:22.149613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:22.149852 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:22.149929 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:22.149932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.150860 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.150863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:22.151529 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:22.151531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.152018 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.152019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:22.152073 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.152810 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:22.152820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:22.155780 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
W20260928 15:03:22.156076 2873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:22.149216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:56541 (local address 127.25.123.62:45537)
0928 15:03:22.149349 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:22.149350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:22.149380 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:03:22.149413 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:22.149414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:22.149425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:22.149479 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:22.150088 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.150717 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.151673 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.151912 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.151988 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.155874 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:56541: BlockingRecv error: recv got EOF from 127.25.123.1:56541 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
Sep 28 15:03:22 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:22 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:22.380982 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.373328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:33119)
0928 15:03:22.373566 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.373579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:22.373656 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:22.373969 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:22.373971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:22.374244 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:22.374324 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:22.374327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.375197 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.375199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:22.375839 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:22.375842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.376548 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.376549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:22.376610 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.377373 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:22.377387 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:22.380892 (+ 3505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
W20260928 15:03:22.381281 2873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:22.373482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:33119 (local address 127.25.123.62:45537)
0928 15:03:22.373544 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:22.373546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:22.373686 (+ 140us) server_negotiation.cc:407] Connection header received
0928 15:03:22.373745 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:22.373746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:22.373761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:22.373850 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:22.374479 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.375047 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.376011 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.376297 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.376510 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.381116 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:33119: BlockingRecv error: recv got EOF from 127.25.123.2:33119 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 15:03:22 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:22 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:22.600172 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:22.594297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:35937)
0928 15:03:22.594485 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:22.594506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:22.594568 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:22.595067 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:22.595068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:22.595312 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:22.595369 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:22.595371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.596329 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.596330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:22.596993 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:22.596995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.597456 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.597457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:22.597503 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.598195 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:22.598200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:22.600108 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
W20260928 15:03:22.600417 2873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:22.594783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:35937 (local address 127.25.123.62:45537)
0928 15:03:22.594879 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:22.594881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:22.594892 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:22.594930 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:22.594931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:22.594944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:22.595012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:22.595483 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.596166 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.597118 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:22.597361 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:22.597428 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:22.600320 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:35937: BlockingRecv error: recv got EOF from 127.25.123.3:35937 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 15:03:23 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:23 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:23.165341 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:23.156753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:48809)
0928 15:03:23.157043 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:23.157065 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:23.157381 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:23.157728 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:23.157729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:23.158154 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:23.158215 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:23.158218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.159224 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.159225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:23.159771 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:23.159773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.160641 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.160642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:23.160698 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:23.162396 (+ 1698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:23.162403 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:23.165272 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":138,"threads_started":1}
W20260928 15:03:23.165593 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:23.156989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:48809 (local address 127.25.123.62:45537)
0928 15:03:23.157436 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:23.157439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:23.157459 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:03:23.157494 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:23.157495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:23.157511 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:23.157624 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:23.158359 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.158934 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.160136 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.160477 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.160560 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:23.165518 (+ 4958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:48809: BlockingRecv error: recv got EOF from 127.25.123.1:48809 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":292,"threads_started":1}
Sep 28 15:03:23 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:23 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:23.389803 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:23.382160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:53681)
0928 15:03:23.382309 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:23.382325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:23.382397 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:23.382864 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:23.382867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:23.383265 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:23.383355 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:23.383359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.384305 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.384308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:23.385092 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:23.385095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.385636 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.385638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:23.385708 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:23.386610 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:23.386617 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:23.389682 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260928 15:03:23.390040 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:23.382385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:53681 (local address 127.25.123.62:45537)
0928 15:03:23.382547 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:23.382550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:23.382562 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:23.382598 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:23.382599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:23.382654 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:23.382737 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:23.383512 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.384034 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.385270 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.385489 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.385573 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:23.389978 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:53681: BlockingRecv error: recv got EOF from 127.25.123.2:53681 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 28 15:03:23 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:23 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:23.613550 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:23.600950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:32871)
0928 15:03:23.601162 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:23.601175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:23.601266 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:23.601526 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:23.601528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:23.601758 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:23.601810 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:23.601812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.602551 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.602552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:23.603152 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:23.603154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.603543 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.603544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:23.603586 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:23.611486 (+ 7900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:23.611494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:23.613484 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":120,"threads_started":1}
W20260928 15:03:23.613758 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:23.601125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:32871 (local address 127.25.123.62:45537)
0928 15:03:23.601209 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:23.601211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:23.601230 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:03:23.601374 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:23.601375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:23.601389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:23.601451 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:23.601909 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.602444 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.603265 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:23.603462 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:23.603535 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:23.613685 (+ 10150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:32871: BlockingRecv error: recv got EOF from 127.25.123.3:32871 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 15:03:24 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:24 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:24.171877 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.166172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:51081)
0928 15:03:24.166391 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:24.166403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:24.166458 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:24.166668 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:24.166669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:24.166901 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:24.166951 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.166953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.167885 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.167887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.168512 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.168515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.168963 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.168964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.169013 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.169698 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:24.169704 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:24.171811 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":66,"threads_started":1}
W20260928 15:03:24.172227 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:24.166353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:51081 (local address 127.25.123.62:45537)
0928 15:03:24.166476 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:24.166478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:24.166487 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:03:24.166565 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:24.166566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:24.166579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:24.166653 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:24.167033 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.167729 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.168595 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.168813 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.168938 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.172163 (+ 3225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:51081: BlockingRecv error: recv got EOF from 127.25.123.1:51081 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
Sep 28 15:03:24 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:24 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:24.397652 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.390582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:57039)
0928 15:03:24.390830 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:24.390841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:24.390922 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:24.391294 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:24.391296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:24.391496 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:24.391557 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.391559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.392754 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.392758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.393381 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.393383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.393879 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.393882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.394011 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.394932 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:24.394938 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:24.397584 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":175,"threads_started":1}
W20260928 15:03:24.397819 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:24.390963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:57039 (local address 127.25.123.62:45537)
0928 15:03:24.391097 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:24.391099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:24.391110 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:24.391140 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:24.391141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:24.391152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:24.391218 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:24.391745 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.392590 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.393494 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.393704 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.393772 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.397736 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:57039: BlockingRecv error: recv got EOF from 127.25.123.2:57039 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 28 15:03:24 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:24 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:24.620375 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:24.614125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:53585)
0928 15:03:24.614353 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:24.614366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:24.614443 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:24.614750 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:24.614751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:24.614952 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:24.615030 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.615032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.615933 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.615935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.616740 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:24.616745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.617311 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.617313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:24.617379 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.618028 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:24.618052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:24.620276 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
W20260928 15:03:24.620477 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:24.614261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:53585 (local address 127.25.123.62:45537)
0928 15:03:24.614326 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:24.614328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:24.614523 (+ 195us) server_negotiation.cc:407] Connection header received
0928 15:03:24.614578 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:24.614579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:24.614603 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:24.614699 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:24.615201 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.615780 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.616868 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:24.617124 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:24.617218 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:24.620377 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:53585: BlockingRecv error: recv got EOF from 127.25.123.3:53585 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 15:03:25 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:25 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:25.178665 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.172552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:52159)
0928 15:03:25.172697 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:25.172712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:25.172874 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:25.173125 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:25.173128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:25.173389 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:25.173602 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.173606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.174410 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.174411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.175250 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.175253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.175785 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.175786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.175877 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.176502 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:25.176508 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:25.178554 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
W20260928 15:03:25.178753 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:25.172679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:52159 (local address 127.25.123.62:45537)
0928 15:03:25.172819 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:25.172821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:25.172906 (+ 85us) server_negotiation.cc:407] Connection header received
0928 15:03:25.172935 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:25.172936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:25.172947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:25.173010 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:25.173708 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.174278 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.175345 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.175674 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.175783 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.178665 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:52159: BlockingRecv error: recv got EOF from 127.25.123.1:52159 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
Sep 28 15:03:25 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:25 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:25.406096 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.398905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:44421)
0928 15:03:25.399253 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:25.399267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:25.399350 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:25.399725 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:25.399726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:25.399918 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:25.399982 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.399984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.400914 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.400917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.401773 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.401778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.402466 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.402467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.402520 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.403145 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:25.403152 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:25.405993 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
W20260928 15:03:25.406167 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:25.399017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:44421 (local address 127.25.123.62:45537)
0928 15:03:25.399125 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:25.399128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:25.399413 (+ 285us) server_negotiation.cc:407] Connection header received
0928 15:03:25.399450 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:25.399451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:25.399466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:25.399538 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:25.400104 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.400757 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.401950 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.402314 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.402415 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.406104 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:44421: BlockingRecv error: recv got EOF from 127.25.123.2:44421 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 15:03:25 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:25 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:25.627460 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:25.621080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:45911)
0928 15:03:25.621601 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:25.621619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:25.621707 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:25.622058 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:25.622060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:25.622392 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:25.622469 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.622529 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.623577 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.623578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.624146 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:25.624148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.624650 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.624651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:25.624767 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.625326 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:25.625331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:25.627393 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":353,"threads_started":1}
W20260928 15:03:25.627637 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:25.621541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:45911 (local address 127.25.123.62:45537)
0928 15:03:25.621630 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:25.621632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:25.621696 (+ 64us) server_negotiation.cc:407] Connection header received
0928 15:03:25.621911 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:25.621912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:25.621929 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:25.622014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:25.622774 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.623449 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.624283 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:25.624555 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:25.624618 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:25.627563 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:45911: BlockingRecv error: recv got EOF from 127.25.123.3:45911 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 15:03:26 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:26 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:26.185086 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.179408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:35299)
0928 15:03:26.179574 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.179594 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.179650 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.179940 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.179942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.180115 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:26.180167 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.180168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.181038 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.181039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.181581 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.181583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.181991 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.181992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.182046 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.182698 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:26.182703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:26.184971 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
W20260928 15:03:26.185149 2889 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:26.179559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:35299 (local address 127.25.123.62:45537)
0928 15:03:26.179714 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.179717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.179728 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:26.179809 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.179810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.179826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.179890 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:26.180283 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.180935 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.181678 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.181889 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.181955 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.185075 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:35299: BlockingRecv error: recv got EOF from 127.25.123.1:35299 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":171,"threads_started":1}
Sep 28 15:03:26 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:26 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:26.413599 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.406828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:35931)
0928 15:03:26.407050 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.407063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.407140 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.407314 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.407314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.407489 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:26.407538 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.407539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.408329 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.408330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.408945 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.408948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.409783 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.409785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.409861 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.410863 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:26.410870 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:26.413519 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":127,"threads_started":1}
W20260928 15:03:26.413945 2889 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:26.406994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:35931 (local address 127.25.123.62:45537)
0928 15:03:26.407052 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.407053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.407147 (+ 94us) server_negotiation.cc:407] Connection header received
0928 15:03:26.407192 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.407193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.407205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.407260 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:26.407640 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.408201 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.409067 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.409604 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.409683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.413895 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:35931: BlockingRecv error: recv got EOF from 127.25.123.2:35931 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 15:03:26 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:26 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:26.634081 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:26.628280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:59897)
0928 15:03:26.628713 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:26.628732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:26.628834 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:26.629046 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:26.629047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:26.629490 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:26.629543 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.629546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.630477 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.630486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.631034 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:26.631035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.631428 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.631429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:26.631485 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.632069 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:26.632074 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:26.633977 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":363,"threads_started":1}
W20260928 15:03:26.634152 2889 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:26.628695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:59897 (local address 127.25.123.62:45537)
0928 15:03:26.628905 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:26.628907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:26.628917 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:26.628953 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:26.628955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:26.628966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:26.629039 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:26.629870 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.630386 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.631144 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:26.631338 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:26.631427 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:26.634077 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:59897: BlockingRecv error: recv got EOF from 127.25.123.3:59897 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Sep 28 15:03:27 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:27 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:27.193501 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.185890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:47317)
0928 15:03:27.186286 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:27.186304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:27.186389 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:27.186659 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:27.186661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:27.186913 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:27.186997 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.187000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.188024 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.188026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.189138 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.189142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.189772 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.189774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.189856 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.190735 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:27.190742 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:27.193344 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":144,"threads_started":1}
W20260928 15:03:27.193660 2893 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:27.186084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:47317 (local address 127.25.123.62:45537)
0928 15:03:27.186424 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.186428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.186441 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:27.186476 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.186477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.186489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.186568 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:27.187163 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.187817 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.189369 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.189617 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.189701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.193566 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:47317: BlockingRecv error: recv got EOF from 127.25.123.1:47317 (error 108)
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":259,"threads_started":1}
Sep 28 15:03:27 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:27 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:27.421239 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.414324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:48575)
0928 15:03:27.414531 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:27.414546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:27.414664 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:27.414838 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:27.414840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:27.415044 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:27.415090 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.415092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.416012 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.416014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.416821 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.416823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.417199 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.417200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.417241 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.417980 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:27.417992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:27.421152 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":113,"threads_started":1}
W20260928 15:03:27.421336 2893 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:27.414456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:48575 (local address 127.25.123.62:45537)
0928 15:03:27.414552 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.414554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.414646 (+ 92us) server_negotiation.cc:407] Connection header received
0928 15:03:27.414765 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.414766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.414779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.414835 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:27.415177 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.415738 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.416889 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.417127 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.417187 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.421273 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:48575: BlockingRecv error: recv got EOF from 127.25.123.2:48575 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:03:27 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:27 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:27.640450 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:27.634829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:60295)
0928 15:03:27.635104 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:27.635118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:27.635272 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:27.635454 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:27.635455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:27.635650 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:27.635697 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.635698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.636433 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.636434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.636978 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:27.636980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.637598 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.637599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:27.637675 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.638265 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:27.638270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:27.640328 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
W20260928 15:03:27.640607 2893 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:27.634975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:60295 (local address 127.25.123.62:45537)
0928 15:03:27.635104 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:27.635107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:27.635213 (+ 106us) server_negotiation.cc:407] Connection header received
0928 15:03:27.635356 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:27.635358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:27.635371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:27.635446 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:27.635785 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.636306 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.637139 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:27.637439 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:27.637500 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:27.640514 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:60295: BlockingRecv error: recv got EOF from 127.25.123.3:60295 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 15:03:28 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:28 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:28.200762 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:28.194180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:39807)
0928 15:03:28.194312 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:28.194331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:28.194386 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:28.194752 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:28.194754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:28.195118 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:28.195197 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:28.195201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.196262 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.196264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:28.196973 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:28.196976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.197431 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.197432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:28.197497 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:28.198088 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:28.198094 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:28.200645 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
W20260928 15:03:28.200966 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:28.194315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:39807 (local address 127.25.123.62:45537)
0928 15:03:28.194494 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:28.194496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:28.194508 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:28.194548 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:28.194549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:28.194560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:28.194648 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:28.195391 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.196113 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.197101 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.197330 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.197429 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:28.200839 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:39807: BlockingRecv error: recv got EOF from 127.25.123.1:39807 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":115,"threads_started":1}
Sep 28 15:03:28 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:28 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:28.428614 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:28.422189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:55163)
0928 15:03:28.422421 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:28.422436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:28.422512 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:28.422887 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:28.422889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:28.423082 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:28.423144 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:28.423146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.424236 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.424237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:28.425105 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:28.425108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.425544 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.425545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:28.425614 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:28.426333 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:28.426339 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:28.428536 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
W20260928 15:03:28.428872 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:28.422389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:55163 (local address 127.25.123.62:45537)
0928 15:03:28.422596 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:28.422599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:28.422615 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:03:28.422685 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:28.422686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:28.422704 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:28.422794 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:28.423289 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.424116 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.425226 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.425439 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.425518 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:28.428760 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:55163: BlockingRecv error: recv got EOF from 127.25.123.2:55163 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 15:03:28 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:28 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:28.652526 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:28.641274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:33127)
0928 15:03:28.641557 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:28.641580 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:28.641705 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:28.641969 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:28.641970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:28.642175 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:28.642222 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:28.642224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.642929 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.642931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:28.643580 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:28.643583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.644053 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.644054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:28.644112 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:28.650363 (+ 6251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:28.650370 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:28.652438 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":133,"threads_started":1}
W20260928 15:03:28.652630 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:28.641435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:33127 (local address 127.25.123.62:45537)
0928 15:03:28.641558 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:28.641560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:28.641678 (+ 118us) server_negotiation.cc:407] Connection header received
0928 15:03:28.641810 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:28.641812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:28.641888 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:28.641960 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:28.642308 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.642834 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.643737 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:28.643954 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:28.644049 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:28.652560 (+ 8511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:33127: BlockingRecv error: recv got EOF from 127.25.123.3:33127 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 15:03:29 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:29 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:29.208981 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:29.201544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:33443)
0928 15:03:29.201832 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:29.201855 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:29.201934 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:29.202366 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:29.202366 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:29.202548 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:29.202600 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:29.202601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.203474 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.203476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:29.204467 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:29.204470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.204976 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.204978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:29.205062 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:29.205953 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:29.205959 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:29.208789 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
W20260928 15:03:29.209043 2901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:29.201878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:33443 (local address 127.25.123.62:45537)
0928 15:03:29.202191 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:29.202193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:29.202205 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:29.202237 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:29.202237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:29.202249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:29.202319 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:29.202731 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.203359 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.204585 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.204822 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.204895 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:29.208975 (+ 4080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:33443: BlockingRecv error: recv got EOF from 127.25.123.1:33443 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":255,"threads_started":1}
Sep 28 15:03:29 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:29 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:29.435545 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:29.429128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:51239)
0928 15:03:29.429392 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:29.429406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:29.429526 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:29.429817 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:29.429819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:29.430041 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:29.430105 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:29.430107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.431177 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.431178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:29.432046 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:29.432050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.432611 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.432612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:29.432671 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:29.433228 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:29.433234 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:29.435448 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
W20260928 15:03:29.435635 2901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:29.429252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:51239 (local address 127.25.123.62:45537)
0928 15:03:29.429398 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:29.429401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:29.429490 (+ 89us) server_negotiation.cc:407] Connection header received
0928 15:03:29.429645 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:29.429646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:29.429661 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:29.429724 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:29.430239 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.431057 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.432149 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.432519 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.432596 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:29.435572 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:51239: BlockingRecv error: recv got EOF from 127.25.123.2:51239 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 15:03:29 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:29 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:29.660636 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:29.653251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:42041)
0928 15:03:29.653528 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:29.653544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:29.653647 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:29.654145 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:29.654148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:29.654608 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:29.654681 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:29.654684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.655977 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.655979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:29.656779 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:29.656782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.657271 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.657272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:29.657332 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:29.658105 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:29.658112 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:29.660554 (+ 2442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":170,"threads_started":1}
W20260928 15:03:29.660787 2901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:29.653437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:42041 (local address 127.25.123.62:45537)
0928 15:03:29.653534 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:29.653537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:29.653626 (+ 89us) server_negotiation.cc:407] Connection header received
0928 15:03:29.653805 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:29.653805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:29.653929 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:29.654099 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:29.654876 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.655818 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.656889 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:29.657157 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:29.657269 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:29.660702 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:42041: BlockingRecv error: recv got EOF from 127.25.123.3:42041 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 15:03:30 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:30 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:30.215279 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:30.209737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:56245)
0928 15:03:30.210004 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:30.210016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:30.210075 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:30.210218 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:30.210219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:30.210418 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:30.210467 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:30.210468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.211451 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.211453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:30.212047 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:30.212049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.212526 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.212527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:30.212572 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:30.213055 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:30.213060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:30.215168 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":185,"threads_started":1}
W20260928 15:03:30.215355 2905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:30.209913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:56245 (local address 127.25.123.62:45537)
0928 15:03:30.210043 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:30.210045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:30.210058 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:30.210139 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:30.210140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:30.210151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:30.210212 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:30.210599 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.211228 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.212159 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.212383 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.212523 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:30.215279 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:56245: BlockingRecv error: recv got EOF from 127.25.123.1:56245 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
Sep 28 15:03:30 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:30 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:30.442245 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:30.436258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:53159)
0928 15:03:30.436466 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:30.436481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:30.436552 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:30.436722 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:30.436723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:30.436933 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:30.436981 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:30.436983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.437686 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.437687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:30.438439 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:30.438443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.438873 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.438875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:30.438937 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:30.439499 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:30.439503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:30.442131 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":124,"threads_started":1}
W20260928 15:03:30.442410 2905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:30.436443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:53159 (local address 127.25.123.62:45537)
0928 15:03:30.436538 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:30.436541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:30.436552 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:30.436621 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:30.436622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:30.436637 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:30.436712 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:30.437078 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.437584 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.438570 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.438774 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.438872 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:30.442318 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:53159: BlockingRecv error: recv got EOF from 127.25.123.2:53159 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 15:03:30 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:30 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.3@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:30.668023 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:30.661461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.3:39833)
0928 15:03:30.661770 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:30.661791 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:30.662008 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:30.662191 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:30.662192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:30.662464 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:30.662534 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:30.662537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.663367 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.663368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:30.664163 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:30.664169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.664798 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.664799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:30.664863 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:30.665519 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:30.665524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:30.667929 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":62,"threads_started":1}
W20260928 15:03:30.668113 2905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:30.661905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:39833 (local address 127.25.123.62:45537)
0928 15:03:30.662000 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:30.662003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:30.662016 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:30.662094 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:30.662096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:30.662115 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:30.662174 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:30.662642 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.663257 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.664367 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:30.664661 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:30.664792 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:30.668047 (+ 3255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.3:39833: BlockingRecv error: recv got EOF from 127.25.123.3:39833 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 15:03:31 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:31 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.1@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:31.221238 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:31.216029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.1:52827)
0928 15:03:31.216182 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:31.216192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:31.216244 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:31.216439 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:31.216439 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:31.216611 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:31.216664 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:31.216665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.217474 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.217474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:31.218097 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:31.218099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.218490 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.218491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:31.218546 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.219138 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:31.219143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:31.221161 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
W20260928 15:03:31.221472 2909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:31.216105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:52827 (local address 127.25.123.62:45537)
0928 15:03:31.216281 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:31.216283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:31.216293 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:03:31.216324 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:31.216325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:31.216336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:31.216425 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:31.216786 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.217367 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.218184 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.218408 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.218491 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.221390 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.1:52827: BlockingRecv error: recv got EOF from 127.25.123.1:52827 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":117,"threads_started":1}
Sep 28 15:03:31 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:03:31 dist-test-slave-qnz0 krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.123.2@KRBTEST.COM for oryx/127.25.123.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:03:31.449321 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:31.443319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45537 (local address 127.25.123.2:49525)
0928 15:03:31.443539 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:31.443552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:31.443629 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:31.443875 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:31.443876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:31.444115 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:31.444202 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:31.444205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.445137 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.445139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:31.445725 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:31.445727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.446139 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.446140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:31.446188 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.446734 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:31.446739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:31.449212 (+ 2473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.123.62:45537: Server oryx/127.25.123.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
W20260928 15:03:31.449373 2909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:31.443465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:49525 (local address 127.25.123.62:45537)
0928 15:03:31.443559 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:31.443561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:31.443621 (+ 60us) server_negotiation.cc:407] Connection header received
0928 15:03:31.443709 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:31.443711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:31.443725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:31.443788 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:31.444349 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.445020 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.445837 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.446055 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.446119 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.449308 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.123.2:49525: BlockingRecv error: recv got EOF from 127.25.123.2:49525 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:03:31.518857 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 2428
I20260928 15:03:31.523876 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 2582
I20260928 15:03:31.528769 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 2716
I20260928 15:03:31.534041 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 2356
2026-09-28T15:03:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18194 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T15:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:31Z Disabled control of system clock
I20260928 15:03:31.568517 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33507
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:33507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:03:31.667657 2914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:31.667927 2914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:31.667950 2914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:31.669374 2914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:31.669430 2914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:31.669443 2914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:31.669457 2914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:31.669469 2914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:31.670961 2914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:33507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:31.671217 2914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:31.671499 2914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:31.677937 2921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:31.678117 2920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:31.678243 2914 server_base.cc:1034] running on GCE node
W20260928 15:03:31.678411 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:31.678817 2914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:31.679215 2914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:31.680541 2914 hybrid_clock.cc:648] HybridClock initialized: now 1790607811680517 us; error 41 us; skew 500 ppm
I20260928 15:03:31.682312 2914 webserver.cc:460] Webserver started at http://127.25.123.62:38375/ using document root <none> and password file <none>
I20260928 15:03:31.682745 2914 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:31.682835 2914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:31.683000 2914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:31.684710 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "1918422fdb6a432881443eb4a453c7b8"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-qnz0"
server_key: "8af3c321d220886f012e2fc0699debf5"
server_key_iv: "8db1d0a677743a22b41ce57b0bef6d4c"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.685186 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "1918422fdb6a432881443eb4a453c7b8"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-qnz0"
server_key: "8af3c321d220886f012e2fc0699debf5"
server_key_iv: "8db1d0a677743a22b41ce57b0bef6d4c"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.687009 2914 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:03:31.688170 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:31.688547 2914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:31.688701 2914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "1918422fdb6a432881443eb4a453c7b8"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-qnz0"
server_key: "8af3c321d220886f012e2fc0699debf5"
server_key_iv: "8db1d0a677743a22b41ce57b0bef6d4c"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.688802 2914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:31.703663 2914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:31.704314 2914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:31.704583 2914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:31.709836 2914 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:33507
I20260928 15:03:31.709872 2981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:33507 every 8 connection(s)
I20260928 15:03:31.710297 2914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:31.711179 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:31.714826 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8: Bootstrap starting.
I20260928 15:03:31.715472 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 2914
I20260928 15:03:31.715622 2982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:31.715693 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:03:31.715960 2982 log.cc:824] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:31.716006 26092 external_mini_cluster.cc:1428] Setting key a0d9e90bf80aa2452b0405ea43b7c1df
I20260928 15:03:31.717906 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8: No bootstrap required, opened a new log
I20260928 15:03:31.720990 2982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1918422fdb6a432881443eb4a453c7b8" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33507 } }
I20260928 15:03:31.721257 2982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:31.721314 2982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1918422fdb6a432881443eb4a453c7b8, State: Initialized, Role: FOLLOWER
I20260928 15:03:31.721526 2982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1918422fdb6a432881443eb4a453c7b8" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33507 } }
I20260928 15:03:31.721661 2982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:31.721714 2982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:31.721779 2982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:31.722635 2982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1918422fdb6a432881443eb4a453c7b8" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33507 } }
I20260928 15:03:31.723067 2982 leader_election.cc:302] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1918422fdb6a432881443eb4a453c7b8; no voters:
I20260928 15:03:31.723357 2982 leader_election.cc:288] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:31.723368 2987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:31.723606 2987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1918422fdb6a432881443eb4a453c7b8, State: Running, Role: LEADER
I20260928 15:03:31.723775 2982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:31.723802 2987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1918422fdb6a432881443eb4a453c7b8" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33507 } }
I20260928 15:03:31.725208 2988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1918422fdb6a432881443eb4a453c7b8. Latest consensus state: current_term: 1 leader_uuid: "1918422fdb6a432881443eb4a453c7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1918422fdb6a432881443eb4a453c7b8" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33507 } } }
I20260928 15:03:31.725453 2988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:31.725540 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:31.717416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.25.123.62:33507)
0928 15:03:31.717669 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:31.717678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:31.717711 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:03:31.718382 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:31.718391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:31.718469 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:31.718650 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:31.719143 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.720616 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.722375 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.722997 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.723141 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.724050 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:31.724055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:31.724055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:31.724462 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:31.724517 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:31.724537 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:31.724623 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:31.724980 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 15:03:31.725093 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":124,"threads_started":1}
W20260928 15:03:31.725908 3001 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:03:31.726696 3001 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:03:31.725318 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1918422fdb6a432881443eb4a453c7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1918422fdb6a432881443eb4a453c7b8" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33507 } } }
I20260928 15:03:31.726887 3002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:31.726960 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:31.727159 3002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:31.729102 3002 catalog_manager.cc:1347] Generated new cluster ID: 75f8dab1aae24b019838bc89e5516ecd
I20260928 15:03:31.729176 3002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:31.744092 3002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:31.745313 3002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:31.761121 3002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1918422fdb6a432881443eb4a453c7b8: Generated new TSK 0
I20260928 15:03:31.761412 3002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:31.763564 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33507
--builtin_ntp_servers=127.25.123.20:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 15:03:31.873633 3006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:31.873872 3006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:31.873899 3006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:31.875813 3006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:31.875946 3006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:31.875989 3006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:31.878520 3006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:31.879321 3006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:31.879817 3006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:31.886983 3011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:31.886983 3014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:31.887004 3006 server_base.cc:1034] running on GCE node
W20260928 15:03:31.887136 3012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:31.887406 3006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:31.887827 3006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:31.889106 3006 hybrid_clock.cc:648] HybridClock initialized: now 1790607811889041 us; error 66 us; skew 500 ppm
I20260928 15:03:31.891453 3006 webserver.cc:460] Webserver started at http://127.25.123.1:35635/ using document root <none> and password file <none>
I20260928 15:03:31.891791 3006 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:31.891880 3006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:31.892063 3006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:31.893371 3006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "64fea22464eb4a378bddf8e59693a4b2"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-qnz0"
server_key: "35cedecbf9533117da2bdb3cedb720bd"
server_key_iv: "945a4d275b0715221d086fdb33dbd734"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.893837 3006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "64fea22464eb4a378bddf8e59693a4b2"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-qnz0"
server_key: "35cedecbf9533117da2bdb3cedb720bd"
server_key_iv: "945a4d275b0715221d086fdb33dbd734"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.895581 3006 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 15:03:31.896709 3020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:31.897051 3006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:31.897145 3006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "64fea22464eb4a378bddf8e59693a4b2"
format_stamp: "Formatted at 2026-09-28 15:03:31 on dist-test-slave-qnz0"
server_key: "35cedecbf9533117da2bdb3cedb720bd"
server_key_iv: "945a4d275b0715221d086fdb33dbd734"
server_key_version: "encryptionkey@0"
I20260928 15:03:31.897246 3006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:31.929214 3006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:31.929634 3006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:31.929759 3006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:31.930049 3006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:31.930718 3006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:31.930806 3006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:31.930850 3006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:31.930881 3006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:31.940848 3006 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:39675
I20260928 15:03:31.940936 3134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:39675 every 8 connection(s)
I20260928 15:03:31.941464 3006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:31.948881 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:31.942806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:43071 (local address 127.25.123.62:33507)
0928 15:03:31.942919 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:31.942922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:31.943290 (+ 368us) server_negotiation.cc:407] Connection header received
0928 15:03:31.944006 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:31.944008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:31.944023 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:31.944086 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:31.945036 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.945627 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.946818 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.947046 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.947136 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.948538 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:31.948540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:31.948541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:31.948548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:31.948572 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:31.948576 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:31.948645 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:31.948743 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 15:03:31.948778 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:03:31.949152 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:31.942774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33507 (local address 127.25.123.1:43071)
0928 15:03:31.943133 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:31.943181 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:31.943860 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:31.944189 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:31.944194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:31.944209 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:31.944910 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:31.944916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.945757 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.945758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:31.946707 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:31.946709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:31.947161 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:31.947162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:31.947315 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:31.948253 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:31.948254 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:31.948290 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:31.948319 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:31.948646 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:31.948654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:31.948658 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:31.948747 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:03:31.948878 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260928 15:03:31.950078 3135 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33507
I20260928 15:03:31.950196 3135 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:31.950469 3135 heartbeater.cc:511] Master 127.25.123.62:33507 requested a full tablet report, sending...
I20260928 15:03:31.951159 2946 ts_manager.cc:194] Registered new tserver with Master: 64fea22464eb4a378bddf8e59693a4b2 (127.25.123.1:39675)
I20260928 15:03:31.951618 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3006
I20260928 15:03:31.951864 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:31.951943 2946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.1:43071
I20260928 15:03:31.952251 26092 external_mini_cluster.cc:1428] Setting key 1fe4f4e1d3791b3df001f116c79d0a97
I20260928 15:03:31.953791 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33507
--builtin_ntp_servers=127.25.123.20:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 15:03:32.061339 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.061602 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.061630 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.062932 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.062983 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.063009 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:32.064483 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:32.064726 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.065093 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.070935 3143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.071928 3146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.071990 3144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.072736 3138 server_base.cc:1034] running on GCE node
I20260928 15:03:32.073176 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.074445 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.078011 3138 hybrid_clock.cc:648] HybridClock initialized: now 1790607812077879 us; error 152 us; skew 500 ppm
I20260928 15:03:32.080336 3138 webserver.cc:460] Webserver started at http://127.25.123.2:36595/ using document root <none> and password file <none>
I20260928 15:03:32.080756 3138 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.080856 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.081048 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.083019 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "d02dfc03ee084684a2b3d620883930c4"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "908069b0f3bec37181dba4626c1abe4a"
server_key_iv: "b72186c8a4f7e9396b5815b282d369dd"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.083745 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "d02dfc03ee084684a2b3d620883930c4"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "908069b0f3bec37181dba4626c1abe4a"
server_key_iv: "b72186c8a4f7e9396b5815b282d369dd"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.085697 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.005s
I20260928 15:03:32.086869 3152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.087229 3138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:03:32.087363 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "d02dfc03ee084684a2b3d620883930c4"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "908069b0f3bec37181dba4626c1abe4a"
server_key_iv: "b72186c8a4f7e9396b5815b282d369dd"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.087467 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.094782 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.095161 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.095326 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.095575 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:32.095944 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:32.096004 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.096045 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:32.096073 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.102627 3138 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:34851
I20260928 15:03:32.102674 3266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:34851 every 8 connection(s)
I20260928 15:03:32.103101 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:32.109951 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.104052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:54793 (local address 127.25.123.62:33507)
0928 15:03:32.104204 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.104207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.104293 (+ 86us) server_negotiation.cc:407] Connection header received
0928 15:03:32.104993 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.104994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.105006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.105067 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.106034 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.106869 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.108059 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.108276 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.108341 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.109423 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.109426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.109426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:32.109435 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.109462 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:32.109468 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.109631 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.109848 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.109879 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:03:32.110102 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.103946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33507 (local address 127.25.123.2:54793)
0928 15:03:32.104149 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:32.104201 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:32.104853 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:32.105146 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:32.105149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:32.105159 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:32.105730 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.105735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.106989 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.106990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.107960 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.107964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.108458 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.108460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.108612 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.109203 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:32.109203 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:32.109240 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:32.109265 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:32.109663 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:32.109668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:32.109671 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:03:32.109747 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 15:03:32.109883 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260928 15:03:32.110695 3267 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33507
I20260928 15:03:32.110785 3267 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:32.111006 3267 heartbeater.cc:511] Master 127.25.123.62:33507 requested a full tablet report, sending...
I20260928 15:03:32.111483 2946 ts_manager.cc:194] Registered new tserver with Master: d02dfc03ee084684a2b3d620883930c4 (127.25.123.2:34851)
I20260928 15:03:32.111563 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3138
I20260928 15:03:32.111641 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:32.111846 26092 external_mini_cluster.cc:1428] Setting key baaa439ad994e95babf18e4846309460
I20260928 15:03:32.112016 2946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.2:54793
I20260928 15:03:32.112884 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33507
--builtin_ntp_servers=127.25.123.20:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 15:03:32.216547 3270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.216900 3270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.216931 3270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.219686 3270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.219769 3270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.219825 3270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:32.221635 3270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:32.221995 3270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.222362 3270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.230397 3276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.230397 3278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.230422 3275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.231089 3270 server_base.cc:1034] running on GCE node
I20260928 15:03:32.231328 3270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.231628 3270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.233033 3270 hybrid_clock.cc:648] HybridClock initialized: now 1790607812232995 us; error 43 us; skew 500 ppm
I20260928 15:03:32.235786 3270 webserver.cc:460] Webserver started at http://127.25.123.3:44963/ using document root <none> and password file <none>
I20260928 15:03:32.236498 3270 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.236598 3270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.236779 3270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.238148 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "8a20417a47714d4f919d4a79e053255e"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "89fd34f02b5dc7c5339f1ca5966772fd"
server_key_iv: "e719c1a981fc8b9b9d3d0280a07f5932"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.238598 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "8a20417a47714d4f919d4a79e053255e"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "89fd34f02b5dc7c5339f1ca5966772fd"
server_key_iv: "e719c1a981fc8b9b9d3d0280a07f5932"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.240672 3270 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 15:03:32.242077 3284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.242449 3270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:32.242676 3270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "8a20417a47714d4f919d4a79e053255e"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "89fd34f02b5dc7c5339f1ca5966772fd"
server_key_iv: "e719c1a981fc8b9b9d3d0280a07f5932"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.242825 3270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.267122 3270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.267554 3270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.267725 3270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.268000 3270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:32.268771 3270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:32.268939 3270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.269033 3270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:32.269069 3270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.278169 3270 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:43899
I20260928 15:03:32.278451 3398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:43899 every 8 connection(s)
I20260928 15:03:32.278992 3270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:32.280371 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3270
I20260928 15:03:32.280536 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:32.281267 26092 external_mini_cluster.cc:1428] Setting key a3d71eda0177edef19b5368fbc4d58d7
I20260928 15:03:32.287678 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.280036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:53895 (local address 127.25.123.62:33507)
0928 15:03:32.280103 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.280105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.280414 (+ 309us) server_negotiation.cc:407] Connection header received
0928 15:03:32.281141 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.281142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.281157 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.281257 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.282957 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.283582 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.285342 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.285597 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.285712 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.287264 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.287267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.287268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:32.287276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.287309 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:32.287315 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.287393 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.287522 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.287574 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:03:32.288139 3401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.279850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33507 (local address 127.25.123.3:53895)
0928 15:03:32.280161 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:32.280233 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:32.280988 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:32.281351 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:32.281356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:32.281383 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:32.282656 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.282665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.283714 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.283715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.285138 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:32.285146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:32.285727 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:32.285729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:32.285944 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:32.286830 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:32.286831 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:32.286995 (+ 164us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:32.287063 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:32.287398 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:32.287405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:32.287410 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:32.287513 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:03:32.287706 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":190,"threads_started":1}
I20260928 15:03:32.288910 3399 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33507
I20260928 15:03:32.289024 3399 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:32.289264 3399 heartbeater.cc:511] Master 127.25.123.62:33507 requested a full tablet report, sending...
I20260928 15:03:32.289928 2946 ts_manager.cc:194] Registered new tserver with Master: 8a20417a47714d4f919d4a79e053255e (127.25.123.3:43899)
I20260928 15:03:32.290570 2946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.3:53895
I20260928 15:03:32.295171 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:03:32.298934 2986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:32.297462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.25.123.62:33507)
0928 15:03:32.297990 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.297993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.298011 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:03:32.298048 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.298049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.298065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.298376 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.298620 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39646: BlockingRecv error: recv got EOF from 127.0.0.1:39646 (error 108)
Metrics: {"server-negotiator.queue_time_us":392}
I20260928 15:03:32.299350 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3006
I20260928 15:03:32.306756 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3138
I20260928 15:03:32.313769 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3270
I20260928 15:03:32.319623 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 2914
2026-09-28T15:03:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (784 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T15:03:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:32Z Disabled control of system clock
I20260928 15:03:32.351773 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:36251
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:03:32.446287 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.446543 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.446569 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.447793 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:32.447836 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.447850 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.447863 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:32.447875 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:32.449503 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:36251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:32.449764 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.450040 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.456374 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.456449 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.456377 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.456782 3410 server_base.cc:1034] running on GCE node
I20260928 15:03:32.457052 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.457398 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.458650 3410 hybrid_clock.cc:648] HybridClock initialized: now 1790607812458629 us; error 36 us; skew 500 ppm
I20260928 15:03:32.460274 3410 webserver.cc:460] Webserver started at http://127.25.123.62:36979/ using document root <none> and password file <none>
I20260928 15:03:32.460700 3410 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.460778 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.460940 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.462122 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "2f331aa4128b4b22990e75f524405309"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "c6bcfbc96950a93a521dae6aec161fb8"
server_key_iv: "42d2990a52f0ab0780b5b31f44717321"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.462510 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "2f331aa4128b4b22990e75f524405309"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "c6bcfbc96950a93a521dae6aec161fb8"
server_key_iv: "42d2990a52f0ab0780b5b31f44717321"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.464147 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:03:32.465268 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.465703 3410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:32.465916 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "2f331aa4128b4b22990e75f524405309"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "c6bcfbc96950a93a521dae6aec161fb8"
server_key_iv: "42d2990a52f0ab0780b5b31f44717321"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.466053 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.470363 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.470717 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.470893 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.477577 3410 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:36251
I20260928 15:03:32.477605 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:36251 every 8 connection(s)
I20260928 15:03:32.478052 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:32.479043 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:32.481657 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309: Bootstrap starting.
I20260928 15:03:32.482308 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:32.482638 3477 log.cc:824] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:32.483464 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309: No bootstrap required, opened a new log
I20260928 15:03:32.485289 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f331aa4128b4b22990e75f524405309" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36251 } }
I20260928 15:03:32.485487 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:32.485538 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f331aa4128b4b22990e75f524405309, State: Initialized, Role: FOLLOWER
I20260928 15:03:32.485728 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f331aa4128b4b22990e75f524405309" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36251 } }
I20260928 15:03:32.485821 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:32.485868 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:32.485942 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:32.486606 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f331aa4128b4b22990e75f524405309" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36251 } }
I20260928 15:03:32.486742 3477 leader_election.cc:302] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f331aa4128b4b22990e75f524405309; no voters:
I20260928 15:03:32.486958 3477 leader_election.cc:288] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:32.487232 3480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:32.487437 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:32.487502 3480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [term 1 LEADER]: Becoming Leader. State: Replica: 2f331aa4128b4b22990e75f524405309, State: Running, Role: LEADER
I20260928 15:03:32.487627 3480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f331aa4128b4b22990e75f524405309" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36251 } }
I20260928 15:03:32.487789 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3410
I20260928 15:03:32.487896 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:03:32.488139 26092 external_mini_cluster.cc:1428] Setting key ec96d1e3437a831078378440c63c3592
I20260928 15:03:32.488232 3482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f331aa4128b4b22990e75f524405309. Latest consensus state: current_term: 1 leader_uuid: "2f331aa4128b4b22990e75f524405309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f331aa4128b4b22990e75f524405309" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36251 } } }
I20260928 15:03:32.488232 3481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f331aa4128b4b22990e75f524405309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f331aa4128b4b22990e75f524405309" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 36251 } } }
I20260928 15:03:32.488343 3482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:32.488452 3481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:32.488695 3490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:32.489238 3490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:32.491528 3490 catalog_manager.cc:1347] Generated new cluster ID: 5859b3b73d554135b74cab40ec47ffcd
I20260928 15:03:32.491612 3490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:03:32.491768 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.488913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.25.123.62:36251)
0928 15:03:32.489338 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.489349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.489383 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:03:32.489526 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.489534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.489587 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.489788 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.490962 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.490965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.490965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:32.490970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.491011 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:32.491023 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.491214 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.491351 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.491521 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":75,"threads_started":1}
I20260928 15:03:32.514293 3490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:32.514533 3490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:32.518365 3490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f331aa4128b4b22990e75f524405309: Generated new TSK 0
I20260928 15:03:32.518661 3490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:03:32.523888 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36251
--builtin_ntp_servers=127.25.123.20:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:03:32.638983 3502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.639230 3502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.639258 3502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.640864 3502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.640931 3502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.640964 3502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:32.642549 3502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:32.642879 3502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.643244 3502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.649435 3508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.649487 3510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.649441 3507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.650645 3502 server_base.cc:1034] running on GCE node
I20260928 15:03:32.650974 3502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.651366 3502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.652539 3502 hybrid_clock.cc:648] HybridClock initialized: now 1790607812652500 us; error 73 us; skew 500 ppm
I20260928 15:03:32.654376 3502 webserver.cc:460] Webserver started at http://127.25.123.1:43891/ using document root <none> and password file <none>
I20260928 15:03:32.654685 3502 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.654749 3502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.654888 3502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.656472 3502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "a308552ebec942d08d0e2d0280e70e7f"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "b4365876c50e0f9b9e7595480b311c54"
server_key_iv: "4117f2365febd8fd4418d9d2d0a5236d"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.656934 3502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "a308552ebec942d08d0e2d0280e70e7f"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "b4365876c50e0f9b9e7595480b311c54"
server_key_iv: "4117f2365febd8fd4418d9d2d0a5236d"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.658977 3502 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:03:32.660316 3516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.660544 3502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:32.660640 3502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "a308552ebec942d08d0e2d0280e70e7f"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "b4365876c50e0f9b9e7595480b311c54"
server_key_iv: "4117f2365febd8fd4418d9d2d0a5236d"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.660807 3502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.664498 3502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.664827 3502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.665000 3502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.665254 3502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:32.665612 3502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:32.665674 3502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.665719 3502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:32.665747 3502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.673794 3502 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:46817
I20260928 15:03:32.673846 3632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:46817 every 8 connection(s)
I20260928 15:03:32.674216 3502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:32.678052 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.675171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:60277 (local address 127.25.123.62:36251)
0928 15:03:32.675246 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.675249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.675512 (+ 263us) server_negotiation.cc:407] Connection header received
0928 15:03:32.676016 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.676018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.676026 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.676118 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.677678 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.677680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.677680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:32.677686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.677704 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:32.677708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.677774 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.677971 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.677989 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:03:32.678260 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.675142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:36251 (local address 127.25.123.1:60277)
0928 15:03:32.675366 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:32.675410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:32.675913 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:32.676273 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:32.676277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:32.676290 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:32.677481 (+ 1191us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:32.677481 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:32.677516 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:32.677551 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:32.677797 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:32.677802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:32.677805 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:03:32.677948 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 15:03:32.678017 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260928 15:03:32.678884 3633 heartbeater.cc:347] Connected to a master server at 127.25.123.62:36251
I20260928 15:03:32.679091 3633 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:32.679277 3633 heartbeater.cc:511] Master 127.25.123.62:36251 requested a full tablet report, sending...
I20260928 15:03:32.679982 3441 ts_manager.cc:194] Registered new tserver with Master: a308552ebec942d08d0e2d0280e70e7f (127.25.123.1:46817)
I20260928 15:03:32.680840 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3502
I20260928 15:03:32.680938 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:32.681169 26092 external_mini_cluster.cc:1428] Setting key 9e1c725cef2425b1b45fbf62211b367e
I20260928 15:03:32.682225 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36251
--builtin_ntp_servers=127.25.123.20:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:03:32.779639 3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.779891 3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.779917 3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.781291 3636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.781352 3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.781383 3636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:32.782869 3636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:32.783118 3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.783401 3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.789794 3641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.789963 3636 server_base.cc:1034] running on GCE node
W20260928 15:03:32.789898 3642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.789794 3644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.790447 3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.790856 3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.792061 3636 hybrid_clock.cc:648] HybridClock initialized: now 1790607812792016 us; error 63 us; skew 500 ppm
I20260928 15:03:32.793917 3636 webserver.cc:460] Webserver started at http://127.25.123.2:38293/ using document root <none> and password file <none>
I20260928 15:03:32.794234 3636 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.794317 3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.794492 3636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.795728 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "e98b0b2770b44223b39c8d1341a46f64"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "fe2e83d2a03026ffbcb998a59fce811e"
server_key_iv: "153b6e2448b1787bf8c77f6886dd0796"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.796169 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "e98b0b2770b44223b39c8d1341a46f64"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "fe2e83d2a03026ffbcb998a59fce811e"
server_key_iv: "153b6e2448b1787bf8c77f6886dd0796"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.798354 3636 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260928 15:03:32.799934 3650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.800571 3636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:03:32.800807 3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "e98b0b2770b44223b39c8d1341a46f64"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "fe2e83d2a03026ffbcb998a59fce811e"
server_key_iv: "153b6e2448b1787bf8c77f6886dd0796"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.800957 3636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.805939 3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.806608 3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.806836 3636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.807137 3636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:32.807529 3636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:32.807643 3636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.807693 3636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:32.807719 3636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.815291 3636 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:40351
I20260928 15:03:32.815349 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:40351 every 8 connection(s)
I20260928 15:03:32.815781 3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:32.818895 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3636
I20260928 15:03:32.819046 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:32.819259 26092 external_mini_cluster.cc:1428] Setting key d404a9f88a1a0cd59693b28fb5e4ab34
I20260928 15:03:32.819780 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.816896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:38347 (local address 127.25.123.62:36251)
0928 15:03:32.816954 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.816956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.817186 (+ 230us) server_negotiation.cc:407] Connection header received
0928 15:03:32.817706 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.817707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.817714 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.817783 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.819490 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.819492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.819492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:32.819498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.819517 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:32.819522 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.819619 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.819692 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.819708 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:03:32.820206 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.816727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:36251 (local address 127.25.123.2:38347)
0928 15:03:32.817013 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:32.817061 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:32.817576 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:32.817909 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:32.817913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:32.817922 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:32.819227 (+ 1305us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:32.819228 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:32.819276 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:32.819323 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:32.819604 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:32.819610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:32.819614 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:32.819705 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:03:32.819787 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
I20260928 15:03:32.820834 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36251
--builtin_ntp_servers=127.25.123.20:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 15:03:32.820900 3767 heartbeater.cc:347] Connected to a master server at 127.25.123.62:36251
I20260928 15:03:32.820997 3767 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:32.821158 3767 heartbeater.cc:511] Master 127.25.123.62:36251 requested a full tablet report, sending...
I20260928 15:03:32.821597 3441 ts_manager.cc:194] Registered new tserver with Master: e98b0b2770b44223b39c8d1341a46f64 (127.25.123.2:40351)
W20260928 15:03:32.913969 3770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:32.914273 3770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:32.914315 3770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:32.916565 3770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:32.916656 3770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:32.916720 3770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:32.918584 3770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:32.918951 3770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:32.919308 3770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:32.926051 3778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.926038 3775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:32.926038 3776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:32.926608 3770 server_base.cc:1034] running on GCE node
I20260928 15:03:32.926858 3770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:32.927181 3770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:32.928656 3770 hybrid_clock.cc:648] HybridClock initialized: now 1790607812928618 us; error 46 us; skew 500 ppm
I20260928 15:03:32.930850 3770 webserver.cc:460] Webserver started at http://127.25.123.3:33573/ using document root <none> and password file <none>
I20260928 15:03:32.931247 3770 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:32.931346 3770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:32.931562 3770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:32.933197 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "843a5b3ebf054c34a869f0dcd9bf3221"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "03ee9d351d53301d722c6af02ff2c5d2"
server_key_iv: "28816a9cc2c4fa5cb8bb14fb17687950"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.933764 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "843a5b3ebf054c34a869f0dcd9bf3221"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "03ee9d351d53301d722c6af02ff2c5d2"
server_key_iv: "28816a9cc2c4fa5cb8bb14fb17687950"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.935865 3770 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:32.937673 3784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.938304 3770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:03:32.938469 3770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "843a5b3ebf054c34a869f0dcd9bf3221"
format_stamp: "Formatted at 2026-09-28 15:03:32 on dist-test-slave-qnz0"
server_key: "03ee9d351d53301d722c6af02ff2c5d2"
server_key_iv: "28816a9cc2c4fa5cb8bb14fb17687950"
server_key_version: "encryptionkey@0"
I20260928 15:03:32.938575 3770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:32.942605 3770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:32.942900 3770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:32.943007 3770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:32.943186 3770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:32.943543 3770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:32.943602 3770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.943671 3770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:32.943701 3770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:32.950831 3770 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:37793
I20260928 15:03:32.951152 3898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:37793 every 8 connection(s)
I20260928 15:03:32.951333 3770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:32.956565 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.952343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:38873 (local address 127.25.123.62:36251)
0928 15:03:32.952599 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.952602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.952819 (+ 217us) server_negotiation.cc:407] Connection header received
0928 15:03:32.954334 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.954335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.954344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.954419 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.956201 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:32.956203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:32.956204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:03:32.956210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:32.956228 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:03:32.956233 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:32.956318 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:32.956460 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 15:03:32.956482 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260928 15:03:32.956792 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:32.952202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:36251 (local address 127.25.123.3:38873)
0928 15:03:32.952670 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:32.952720 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:32.953898 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:32.954520 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:32.954524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:32.954535 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:32.955908 (+ 1373us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:03:32.955909 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:32.955944 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:03:32.955981 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:32.956315 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:32.956322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:32.956326 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:32.956452 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 15:03:32.956530 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":121,"threads_started":1}
I20260928 15:03:32.957686 3899 heartbeater.cc:347] Connected to a master server at 127.25.123.62:36251
I20260928 15:03:32.957641 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3770
I20260928 15:03:32.957844 3899 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:32.958101 3899 heartbeater.cc:511] Master 127.25.123.62:36251 requested a full tablet report, sending...
I20260928 15:03:32.958004 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:32.958525 26092 external_mini_cluster.cc:1428] Setting key 29c4b71f37791a37580640da05d8eff8
I20260928 15:03:32.958554 3441 ts_manager.cc:194] Registered new tserver with Master: 843a5b3ebf054c34a869f0dcd9bf3221 (127.25.123.3:37793)
I20260928 15:03:32.959810 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:03:32.962641 3498 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:03:32.961602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.25.123.62:36251)
0928 15:03:32.961768 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:32.961771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:32.962110 (+ 339us) server_negotiation.cc:407] Connection header received
0928 15:03:32.962147 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:32.962148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:32.962154 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:32.962211 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:32.962523 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39040: BlockingRecv error: recv got EOF from 127.0.0.1:39040 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 15:03:32.962934 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3502
I20260928 15:03:32.969292 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3636
I20260928 15:03:32.975718 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3770
I20260928 15:03:32.980890 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3410
2026-09-28T15:03:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (660 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:33 dist-test-slave-qnz0 krb5kdc[3910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:33 dist-test-slave-qnz0 krb5kdc[3910](info): set up 2 sockets
Sep 28 15:03:33 dist-test-slave-qnz0 krb5kdc[3910](info): commencing operation
krb5kdc: starting...
W20260928 15:03:34.174117 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:03:34 dist-test-slave-qnz0 krb5kdc[3910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:03:34.330386 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45059
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:34.427623 3923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:34.427876 3923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:34.427902 3923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:34.429289 3923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:03:34.429356 3923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:34.429371 3923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:34.429385 3923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:03:34.429399 3923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:03:34.431154 3923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:34.431516 3923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:34.431806 3923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:34.437424 3928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.437444 3929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.437451 3931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:34.437717 3923 server_base.cc:1034] running on GCE node
I20260928 15:03:34.438105 3923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:34.438484 3923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:34.439728 3923 hybrid_clock.cc:648] HybridClock initialized: now 1790607814439673 us; error 63 us; skew 500 ppm
Sep 28 15:03:34 dist-test-slave-qnz0 krb5kdc[3910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:34.451125 3923 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:03:34.451999 3923 webserver.cc:460] Webserver started at http://127.25.123.62:35289/ using document root <none> and password file <none>
I20260928 15:03:34.452293 3923 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:34.452381 3923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:34.452576 3923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:34.453759 3923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "c4036c03eb1f4d1fabf9396f28f3b365"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "b8514370d4261cd2176232a638aec351"
server_key_iv: "c979f44ee7c45dfdfced586c0c8b8392"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.454182 3923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "c4036c03eb1f4d1fabf9396f28f3b365"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "b8514370d4261cd2176232a638aec351"
server_key_iv: "c979f44ee7c45dfdfced586c0c8b8392"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.455780 3923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:03:34.456864 3938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.457273 3923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:34.457386 3923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "c4036c03eb1f4d1fabf9396f28f3b365"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "b8514370d4261cd2176232a638aec351"
server_key_iv: "c979f44ee7c45dfdfced586c0c8b8392"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.457715 3923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:34.473656 3923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:34.476222 3923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:34.476358 3923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:34.480845 3923 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:45059
I20260928 15:03:34.480882 3990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:45059 every 8 connection(s)
I20260928 15:03:34.481268 3923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:03:34.482425 3991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:34.485072 3991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365: Bootstrap starting.
I20260928 15:03:34.485728 3991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:34.486038 3991 log.cc:824] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:34.486979 3991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365: No bootstrap required, opened a new log
I20260928 15:03:34.487345 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 3923
I20260928 15:03:34.487437 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:03:34.487664 26092 external_mini_cluster.cc:1428] Setting key 927b695afe0c36f83d48188c1284e97b
I20260928 15:03:34.490299 3991 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45059 } }
I20260928 15:03:34.490511 3991 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:34.490556 3991 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4036c03eb1f4d1fabf9396f28f3b365, State: Initialized, Role: FOLLOWER
I20260928 15:03:34.490741 3991 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45059 } }
I20260928 15:03:34.490849 3991 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:03:34.490886 3991 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:03:34.490937 3991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:34.491832 3991 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45059 } }
I20260928 15:03:34.491957 3991 leader_election.cc:302] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4036c03eb1f4d1fabf9396f28f3b365; no voters:
I20260928 15:03:34.492139 3991 leader_election.cc:288] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:03:34.492261 3996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:34.492447 3996 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [term 1 LEADER]: Becoming Leader. State: Replica: c4036c03eb1f4d1fabf9396f28f3b365, State: Running, Role: LEADER
I20260928 15:03:34.492539 3996 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45059 } }
I20260928 15:03:34.492576 3991 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:03:34.492916 3997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4036c03eb1f4d1fabf9396f28f3b365. Latest consensus state: current_term: 1 leader_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45059 } } }
I20260928 15:03:34.493040 3997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:34.492918 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4036c03eb1f4d1fabf9396f28f3b365" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45059 } } }
I20260928 15:03:34.493302 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365 [sys.catalog]: This master's current role is: LEADER
I20260928 15:03:34.493458 4002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:03:34.493983 4002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:03:34.496541 4002 catalog_manager.cc:1347] Generated new cluster ID: 98c72cc778a34b48849646996256ddcf
I20260928 15:03:34.496630 4002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:03:34 dist-test-slave-qnz0 krb5kdc[3910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:34.503010 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.488369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.25.123.62:45059)
0928 15:03:34.488631 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:34.488644 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:34.488682 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:03:34.489270 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:34.489280 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:34.489363 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:34.489550 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:34.490687 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.492258 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.493993 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.494531 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.494672 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.497997 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:34.498002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:34.498002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:34.498017 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:34.500231 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:34.500818 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:34.500822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:34.500822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:34.500955 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:34.501194 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:34.501194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:34.501195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:34.501531 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:34.501676 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:34.502329 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 15:03:34.502726 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20260928 15:03:34.516178 4002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:03:34.516918 4002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:03:34.526062 4002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365: Generated new TSK 0
I20260928 15:03:34.526363 4002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:03:34.566102 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45059
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:34.659001 4017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:34.659278 4017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:34.659317 4017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:34.660818 4017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:34.660887 4017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:34.660923 4017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:03:34.662711 4017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:34.663003 4017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:34.663306 4017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:34.669421 4023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.669490 4022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.669745 4025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:34.670192 4017 server_base.cc:1034] running on GCE node
I20260928 15:03:34.670501 4017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:34.670859 4017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:34.672065 4017 hybrid_clock.cc:648] HybridClock initialized: now 1790607814672029 us; error 60 us; skew 500 ppm
Sep 28 15:03:34 dist-test-slave-qnz0 krb5kdc[3910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:34.675549 4017 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:03:34.676205 4017 webserver.cc:460] Webserver started at http://127.25.123.1:42033/ using document root <none> and password file <none>
I20260928 15:03:34.676532 4017 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:34.676620 4017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:34.676797 4017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:34.677965 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "ae084f3ad3418e5b627e4ce86a62de8b"
server_key_iv: "0d4effa1dbae286561ea6480d92c3158"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.678386 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "ae084f3ad3418e5b627e4ce86a62de8b"
server_key_iv: "0d4effa1dbae286561ea6480d92c3158"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.679978 4017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:03:34.681339 4032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.681713 4017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:03:34.681846 4017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "ae084f3ad3418e5b627e4ce86a62de8b"
server_key_iv: "0d4effa1dbae286561ea6480d92c3158"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.681955 4017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:34.712620 4017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:34.717597 4017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:34.717775 4017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:34.718014 4017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:34.718454 4017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:34.718528 4017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.718570 4017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:34.718626 4017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.726601 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:34727 every 8 connection(s)
I20260928 15:03:34.726627 4017 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:34727
I20260928 15:03:34.727198 4017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:03:34.733346 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 4017
I20260928 15:03:34.733554 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:03:34.733939 26092 external_mini_cluster.cc:1428] Setting key 84226510f96ba471485466c24048f4a1
Sep 28 15:03:34 dist-test-slave-qnz0 krb5kdc[3910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:34.740458 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.728477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:58621 (local address 127.25.123.62:45059)
0928 15:03:34.728595 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:34.728597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:34.728881 (+ 284us) server_negotiation.cc:407] Connection header received
0928 15:03:34.729550 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:34.729552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:34.729566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:34.729631 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:34.730772 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.731303 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.732408 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.732643 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.732729 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.736053 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:34.736055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:34.736055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:34.736064 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:34.738129 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:34.738591 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:34.738593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:34.738593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:34.738660 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:34.739383 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:34.739384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:34.739384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:34.739570 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:34.739666 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:34.740108 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 15:03:34.740250 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:03:34.740538 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.728150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45059 (local address 127.25.123.1:58621)
0928 15:03:34.728474 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:34.728749 (+ 275us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:34.729408 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:34.729769 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:34.729773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:34.730406 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:34.730625 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.730630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.731413 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.731415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.732278 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.732281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.732753 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.732754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.732890 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.733645 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:34.733657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:34.735919 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:34.738243 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:34.738252 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:34.738254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:34.738505 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:34.738809 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:34.738812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:34.738812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:34.739075 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:34.739735 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:34.739743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:34.739933 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 15:03:34.740051 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:03:34.740241 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260928 15:03:34.741286 4149 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45059
I20260928 15:03:34.741397 4149 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:34.741570 4149 heartbeater.cc:511] Master 127.25.123.62:45059 requested a full tablet report, sending...
I20260928 15:03:34.742162 3955 ts_manager.cc:194] Registered new tserver with Master: 2f8d90f5854f4d9fa27ae9034db1a8f9 (127.25.123.1:34727)
I20260928 15:03:34.742859 3955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:58621
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:03:34.774399 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45059
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:34.870136 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:34.870463 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:34.870510 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:34.872531 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:34.872710 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:34.872769 4154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:03:34.874851 4154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:34.875223 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:34.875535 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:34.882442 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.882439 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:34.882431 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:34.882647 4154 server_base.cc:1034] running on GCE node
I20260928 15:03:34.882951 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:34.883306 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:34.884507 4154 hybrid_clock.cc:648] HybridClock initialized: now 1790607814884500 us; error 52 us; skew 500 ppm
Sep 28 15:03:34 dist-test-slave-qnz0 krb5kdc[3910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:34.887992 4154 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:03:34.888847 4154 webserver.cc:460] Webserver started at http://127.25.123.2:41003/ using document root <none> and password file <none>
I20260928 15:03:34.889345 4154 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:34.889446 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:34.889626 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:34.891064 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "854388a209b14bbaa18f8852a93f97c4"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "746f43d22725e6b281adf6022685a15e"
server_key_iv: "599f33bfe143b50b5abf63d7785cbccb"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.891489 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "854388a209b14bbaa18f8852a93f97c4"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "746f43d22725e6b281adf6022685a15e"
server_key_iv: "599f33bfe143b50b5abf63d7785cbccb"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.893465 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 15:03:34.894536 4169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.894861 4154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:34.894979 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "854388a209b14bbaa18f8852a93f97c4"
format_stamp: "Formatted at 2026-09-28 15:03:34 on dist-test-slave-qnz0"
server_key: "746f43d22725e6b281adf6022685a15e"
server_key_iv: "599f33bfe143b50b5abf63d7785cbccb"
server_key_version: "encryptionkey@0"
I20260928 15:03:34.895053 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:34.921743 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:34.925052 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:34.925197 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:34.925419 4154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:34.925782 4154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:34.925828 4154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.925856 4154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:34.925870 4154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:34.933432 4154 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:46727
I20260928 15:03:34.933478 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:46727 every 8 connection(s)
I20260928 15:03:34.933863 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:03:34.941876 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 4154
I20260928 15:03:34.941973 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:03:34.942359 26092 external_mini_cluster.cc:1428] Setting key 5e4569f80d0fcc98ab87dc280caf8b74
Sep 28 15:03:34 dist-test-slave-qnz0 krb5kdc[3910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607814, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:34.947484 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.934923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:45933 (local address 127.25.123.62:45059)
0928 15:03:34.935037 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:34.935039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:34.935229 (+ 190us) server_negotiation.cc:407] Connection header received
0928 15:03:34.935805 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:34.935806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:34.935818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:34.935884 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:34.937565 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.938356 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.940108 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.940322 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.940471 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.943535 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:34.943537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:34.943537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:34.943545 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:34.945364 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:34.946037 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:34.946039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:34.946039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:34.946111 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:34.946495 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:34.946496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:34.946497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:34.946642 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:34.946718 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:34.947225 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 15:03:34.947335 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:03:34.947510 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:34.934774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45059 (local address 127.25.123.2:45933)
0928 15:03:34.935043 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:34.935095 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:34.935619 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:34.935999 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:34.936003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:34.936729 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:34.937390 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.937398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.938518 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.938519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.939985 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:34.939989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:34.940509 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:34.940510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:34.940653 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:34.941513 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:34.941524 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:34.943403 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:34.945475 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:34.945488 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:34.945492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:34.945920 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:34.946197 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:34.946198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:34.946198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:34.946382 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:34.946748 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:34.946755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:34.947018 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 15:03:34.947148 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 15:03:34.947317 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260928 15:03:34.948097 4285 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45059
I20260928 15:03:34.948220 4285 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:34.948487 4285 heartbeater.cc:511] Master 127.25.123.62:45059 requested a full tablet report, sending...
I20260928 15:03:34.949018 3955 ts_manager.cc:194] Registered new tserver with Master: 854388a209b14bbaa18f8852a93f97c4 (127.25.123.2:46727)
I20260928 15:03:34.949446 3955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:45933
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:03:34.986398 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45059
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:03:35.089088 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:03:35.089424 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:03:35.089504 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:03:35.091023 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:03:35.091209 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:03:35.091274 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:03:35.094314 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:45059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:03:35.094934 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:03:35.095337 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:03:35.102576 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:35.102654 4290 server_base.cc:1034] running on GCE node
W20260928 15:03:35.102626 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:03:35.102625 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:03:35.103814 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:03:35.104172 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:03:35.105381 4290 hybrid_clock.cc:648] HybridClock initialized: now 1790607815105297 us; error 114 us; skew 500 ppm
Sep 28 15:03:35 dist-test-slave-qnz0 krb5kdc[3910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607815, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:03:35.108983 4290 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:03:35.109597 4290 webserver.cc:460] Webserver started at http://127.25.123.3:36063/ using document root <none> and password file <none>
I20260928 15:03:35.109880 4290 fs_manager.cc:359] Metadata directory not provided
I20260928 15:03:35.109938 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:03:35.110086 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:03:35.111276 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "3099b3cdb6034405a591329df5ddbdb4"
format_stamp: "Formatted at 2026-09-28 15:03:35 on dist-test-slave-qnz0"
server_key: "48e26d352622f64f17439e18bcf880d3"
server_key_iv: "c2d74cfc48fbc559ea5e797d1069d6db"
server_key_version: "encryptionkey@0"
I20260928 15:03:35.111668 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "3099b3cdb6034405a591329df5ddbdb4"
format_stamp: "Formatted at 2026-09-28 15:03:35 on dist-test-slave-qnz0"
server_key: "48e26d352622f64f17439e18bcf880d3"
server_key_iv: "c2d74cfc48fbc559ea5e797d1069d6db"
server_key_version: "encryptionkey@0"
I20260928 15:03:35.113458 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:03:35.115265 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.115655 4290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:03:35.115832 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "3099b3cdb6034405a591329df5ddbdb4"
format_stamp: "Formatted at 2026-09-28 15:03:35 on dist-test-slave-qnz0"
server_key: "48e26d352622f64f17439e18bcf880d3"
server_key_iv: "c2d74cfc48fbc559ea5e797d1069d6db"
server_key_version: "encryptionkey@0"
I20260928 15:03:35.115942 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:03:35.137259 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:03:35.140870 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:03:35.141106 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:03:35.141537 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:03:35.141990 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:03:35.142059 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.142102 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:03:35.142132 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:03:35.149731 4290 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:36777
I20260928 15:03:35.149781 4421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:36777 every 8 connection(s)
I20260928 15:03:35.150185 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:03:35.154665 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 4290
I20260928 15:03:35.154789 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:03:35.154989 26092 external_mini_cluster.cc:1428] Setting key 62c8471f0c08dc653d69b43296d2aaf9
Sep 28 15:03:35 dist-test-slave-qnz0 krb5kdc[3910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607815, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:03:35.162194 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.151078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:54421 (local address 127.25.123.62:45059)
0928 15:03:35.151179 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.151182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.151404 (+ 222us) server_negotiation.cc:407] Connection header received
0928 15:03:35.151980 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.151981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.151992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.152055 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.153097 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.153622 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.155061 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.155322 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.155375 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.158278 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:35.158280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:35.158280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:35.158288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:35.160282 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:35.160935 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:35.160936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:35.160937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:35.161011 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:35.161407 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:35.161408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:35.161408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:35.161546 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:35.161608 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.162033 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.162123 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:03:35.162369 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.151025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:45059 (local address 127.25.123.3:54421)
0928 15:03:35.151228 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.151271 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.151857 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.152109 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.152112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.152748 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:03:35.152966 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.152971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.153754 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.153755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.154923 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.154927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.155509 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.155510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.155658 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.156369 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:03:35.156384 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:03:35.158171 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:03:35.160458 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:35.160472 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:35.160475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:35.160816 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:35.161120 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:03:35.161121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:03:35.161121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:03:35.161310 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:03:35.161636 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:03:35.161641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:03:35.161844 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.161947 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.162101 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
I20260928 15:03:35.163177 4422 heartbeater.cc:347] Connected to a master server at 127.25.123.62:45059
I20260928 15:03:35.163372 4422 heartbeater.cc:464] Registering TS with master...
I20260928 15:03:35.163615 4422 heartbeater.cc:511] Master 127.25.123.62:45059 requested a full tablet report, sending...
I20260928 15:03:35.164237 3955 ts_manager.cc:194] Registered new tserver with Master: 3099b3cdb6034405a591329df5ddbdb4 (127.25.123.3:36777)
I20260928 15:03:35.164800 3955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:54421
I20260928 15:03:35.167577 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:03:35.177913 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.169726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.25.123.62:45059)
0928 15:03:35.169801 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.169804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.169838 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:03:35.170001 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.170002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.170015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.170094 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:03:35.171142 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.171679 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.172518 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.172718 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.172811 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.174510 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:03:35.174513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:03:35.174514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:03:35.174526 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:03:35.176508 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:35.176958 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:35.176960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:35.176960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:35.177028 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:03:35.177241 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:03:35.177242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:03:35.177242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:03:35.177426 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:03:35.177542 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.177704 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.177797 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 15:03:35.181061 3955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:03:35.183224 3955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:03:35.192535 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.187393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36777 (local address 127.0.0.1:39752)
0928 15:03:35.187706 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.187716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.187761 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.188219 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.188222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.188224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:35.188331 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.188334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.189773 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.189775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.191016 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.191020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.192106 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.192107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.192209 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.192227 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.192316 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.192412 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":54,"thread_start_us":109,"threads_started":1}
I20260928 15:03:35.192724 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.187393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:46727 (local address 127.0.0.1:48526)
0928 15:03:35.187838 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.187846 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.187911 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.188239 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.188240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.188242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:35.188416 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.188420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.189971 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.189972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.191341 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.191345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.192338 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.192340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.192442 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.192446 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.192541 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.192594 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":267,"thread_start_us":100,"threads_started":1}
I20260928 15:03:35.192939 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.187688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.25.123.3:36777)
0928 15:03:35.188036 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.188040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.188049 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:03:35.188086 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.188088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.188102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.188434 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:35.188502 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.189628 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.191139 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.191990 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.192083 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.192156 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.192809 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.192860 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":77,"threads_started":1}
I20260928 15:03:35.193171 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.187836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.25.123.2:46727)
0928 15:03:35.188034 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.188039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.188050 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:03:35.188094 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.188096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.188112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.188296 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:35.188520 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.189809 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.191444 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.192217 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.192322 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.192415 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.192564 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.193082 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260928 15:03:35.193874 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.187386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:34727 (local address 127.0.0.1:36284)
0928 15:03:35.187563 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.187612 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.187694 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.188218 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.188222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.188227 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:35.188331 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.188334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.189791 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.189792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.191341 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.191345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.193546 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.193547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.193627 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.193631 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.193710 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.193760 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":144,"thread_start_us":117,"threads_started":1}
I20260928 15:03:35.193919 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.187636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.25.123.1:34727)
0928 15:03:35.187952 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.187958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.187971 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:03:35.188020 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.188022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.188040 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.188132 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:35.188443 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.189628 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.192140 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.193443 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.193544 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.193599 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.193774 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.193832 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":56,"threads_started":1}
I20260928 15:03:35.193789 4354 tablet_service.cc:1467] Processing CreateTablet for tablet ae2d7743ea0a4cc8a75eb0998a489747 (DEFAULT_TABLE table=test-table [id=77d3b59a0cd94eae850303f715caa684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:35.194083 4218 tablet_service.cc:1467] Processing CreateTablet for tablet ae2d7743ea0a4cc8a75eb0998a489747 (DEFAULT_TABLE table=test-table [id=77d3b59a0cd94eae850303f715caa684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:35.194378 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d7743ea0a4cc8a75eb0998a489747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:35.194465 4218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d7743ea0a4cc8a75eb0998a489747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:35.194625 4082 tablet_service.cc:1467] Processing CreateTablet for tablet ae2d7743ea0a4cc8a75eb0998a489747 (DEFAULT_TABLE table=test-table [id=77d3b59a0cd94eae850303f715caa684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:03:35.194875 4082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d7743ea0a4cc8a75eb0998a489747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:03:35.197924 4439 tablet_bootstrap.cc:492] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: Bootstrap starting.
I20260928 15:03:35.198169 4441 tablet_bootstrap.cc:492] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Bootstrap starting.
I20260928 15:03:35.198745 4439 tablet_bootstrap.cc:654] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:35.198855 4441 tablet_bootstrap.cc:654] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:35.199455 4439 log.cc:824] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:35.199491 4441 log.cc:824] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:35.199860 4440 tablet_bootstrap.cc:492] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Bootstrap starting.
I20260928 15:03:35.200484 4441 tablet_bootstrap.cc:492] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: No bootstrap required, opened a new log
I20260928 15:03:35.200572 4441 ts_tablet_manager.cc:1397] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:35.200598 4440 tablet_bootstrap.cc:654] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Neither blocks nor log segments found. Creating new log.
I20260928 15:03:35.200685 4439 tablet_bootstrap.cc:492] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: No bootstrap required, opened a new log
I20260928 15:03:35.200765 4439 ts_tablet_manager.cc:1397] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:03:35.201123 4440 log.cc:824] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:03:35.201963 4440 tablet_bootstrap.cc:492] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: No bootstrap required, opened a new log
I20260928 15:03:35.202044 4440 ts_tablet_manager.cc:1397] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:03:35.202615 4439 raft_consensus.cc:348] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.202840 4439 raft_consensus.cc:374] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:35.202888 4439 raft_consensus.cc:729] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d90f5854f4d9fa27ae9034db1a8f9, State: Initialized, Role: FOLLOWER
I20260928 15:03:35.203018 4439 consensus_queue.cc:260] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.202873 4441 raft_consensus.cc:348] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.203100 4441 raft_consensus.cc:374] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:35.203137 4441 raft_consensus.cc:729] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854388a209b14bbaa18f8852a93f97c4, State: Initialized, Role: FOLLOWER
I20260928 15:03:35.203277 4441 consensus_queue.cc:260] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.203483 4439 ts_tablet_manager.cc:1428] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:35.203640 4149 heartbeater.cc:503] Master 127.25.123.62:45059 was elected leader, sending a full tablet report...
I20260928 15:03:35.203570 4441 ts_tablet_manager.cc:1428] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:35.204255 4440 raft_consensus.cc:348] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.204463 4440 raft_consensus.cc:374] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:03:35.204501 4440 raft_consensus.cc:729] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3099b3cdb6034405a591329df5ddbdb4, State: Initialized, Role: FOLLOWER
I20260928 15:03:35.204613 4440 consensus_queue.cc:260] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.204762 4285 heartbeater.cc:503] Master 127.25.123.62:45059 was elected leader, sending a full tablet report...
I20260928 15:03:35.204933 4440 ts_tablet_manager.cc:1428] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:03:35.204968 4422 heartbeater.cc:503] Master 127.25.123.62:45059 was elected leader, sending a full tablet report...
W20260928 15:03:35.228374 4150 tablet.cc:2369] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:03:35.310611 4448 raft_consensus.cc:482] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:35.310770 4448 raft_consensus.cc:504] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.311225 4448 leader_election.cc:288] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8d90f5854f4d9fa27ae9034db1a8f9 (127.25.123.1:34727), 854388a209b14bbaa18f8852a93f97c4 (127.25.123.2:46727)
I20260928 15:03:35.315974 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.311581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:53019 (local address 127.25.123.2:46727)
0928 15:03:35.311632 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.311635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.311647 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:03:35.311834 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.311837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.311930 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.312042 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:35.312518 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.313119 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.314647 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.315094 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.315158 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.315167 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.315832 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.315886 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:03:35.315994 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.311589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:41411 (local address 127.25.123.1:34727)
0928 15:03:35.311635 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.311637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.312072 (+ 435us) server_negotiation.cc:407] Connection header received
0928 15:03:35.312106 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.312107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.312122 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.312219 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:35.312737 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.313265 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.314756 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.315218 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.315475 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.315712 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.315869 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.315920 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:03:35.315995 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.311512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:46727 (local address 127.25.123.3:53019)
0928 15:03:35.311584 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.311597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.311646 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.312101 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.312102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.312105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:35.312244 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.312247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.313240 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.313241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.314511 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.314514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.315203 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.315205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.315436 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.315447 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.315797 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.315882 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260928 15:03:35.316011 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.311524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:34727 (local address 127.25.123.3:41411)
0928 15:03:35.311876 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.311924 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.312042 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.312233 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.312234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.312236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:35.312553 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.312557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.313483 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.313484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.314655 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.314658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.315494 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.315496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.315578 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.315583 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.315847 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.315908 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":2,"thread_start_us":135,"threads_started":1}
I20260928 15:03:35.316570 4239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d7743ea0a4cc8a75eb0998a489747" candidate_uuid: "3099b3cdb6034405a591329df5ddbdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854388a209b14bbaa18f8852a93f97c4" is_pre_election: true
I20260928 15:03:35.317081 4239 raft_consensus.cc:2454] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3099b3cdb6034405a591329df5ddbdb4 in term 0.
I20260928 15:03:35.317095 4103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d7743ea0a4cc8a75eb0998a489747" candidate_uuid: "3099b3cdb6034405a591329df5ddbdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" is_pre_election: true
I20260928 15:03:35.317358 4447 raft_consensus.cc:482] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:03:35.317426 4447 raft_consensus.cc:504] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.317489 4103 raft_consensus.cc:2454] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3099b3cdb6034405a591329df5ddbdb4 in term 0.
I20260928 15:03:35.317740 4306 leader_election.cc:302] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3099b3cdb6034405a591329df5ddbdb4, 854388a209b14bbaa18f8852a93f97c4; no voters:
I20260928 15:03:35.317816 4447 leader_election.cc:288] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8d90f5854f4d9fa27ae9034db1a8f9 (127.25.123.1:34727), 3099b3cdb6034405a591329df5ddbdb4 (127.25.123.3:36777)
I20260928 15:03:35.318056 4448 raft_consensus.cc:2781] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:03:35.318132 4448 raft_consensus.cc:482] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:03:35.318291 4448 raft_consensus.cc:3037] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:35.319429 4448 raft_consensus.cc:504] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.319797 4448 leader_election.cc:288] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [CANDIDATE]: Term 1 election: Requested vote from peers 2f8d90f5854f4d9fa27ae9034db1a8f9 (127.25.123.1:34727), 854388a209b14bbaa18f8852a93f97c4 (127.25.123.2:46727)
I20260928 15:03:35.319972 4239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d7743ea0a4cc8a75eb0998a489747" candidate_uuid: "3099b3cdb6034405a591329df5ddbdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854388a209b14bbaa18f8852a93f97c4"
I20260928 15:03:35.320091 4239 raft_consensus.cc:3037] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:35.320076 4103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d7743ea0a4cc8a75eb0998a489747" candidate_uuid: "3099b3cdb6034405a591329df5ddbdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9"
I20260928 15:03:35.320266 4103 raft_consensus.cc:3037] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:03:35.321069 4239 raft_consensus.cc:2454] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3099b3cdb6034405a591329df5ddbdb4 in term 1.
I20260928 15:03:35.321219 4103 raft_consensus.cc:2454] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3099b3cdb6034405a591329df5ddbdb4 in term 1.
I20260928 15:03:35.321267 4306 leader_election.cc:302] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3099b3cdb6034405a591329df5ddbdb4, 854388a209b14bbaa18f8852a93f97c4; no voters:
I20260928 15:03:35.321421 4448 raft_consensus.cc:2781] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:03:35.321574 4448 raft_consensus.cc:686] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 1 LEADER]: Becoming Leader. State: Replica: 3099b3cdb6034405a591329df5ddbdb4, State: Running, Role: LEADER
I20260928 15:03:35.321684 4448 consensus_queue.cc:237] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } }
I20260928 15:03:35.322492 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.318333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:60289 (local address 127.25.123.1:34727)
0928 15:03:35.318372 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.318374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.318416 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:03:35.318476 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.318477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.318487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.318531 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:35.318735 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.319487 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.321270 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.321818 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.321867 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.321879 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.322361 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.322423 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:03:35.322492 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.318068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36777 (local address 127.25.123.2:58099)
0928 15:03:35.318132 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.318146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.318211 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.318644 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.318645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.318647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:35.318713 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.318714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.319579 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.319580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.321048 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.321051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.322085 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.322085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.322159 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.322162 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.322354 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.322406 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":33}
I20260928 15:03:35.322830 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.318407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:58099 (local address 127.25.123.3:36777)
0928 15:03:35.318456 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.318459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.318468 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:03:35.318493 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.318494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.318505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.318575 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:03:35.318776 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.319487 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.321163 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.321964 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.322041 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.322051 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.322633 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.322761 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 15:03:35.322504 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.318124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:34727 (local address 127.25.123.2:60289)
0928 15:03:35.318365 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.318375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:03:35.318428 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:03:35.318573 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:03:35.318574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:03:35.318575 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:03:35.318643 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.318647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.319743 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.319744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.321152 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:03:35.321154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.321941 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.321942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:03:35.322039 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.322044 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:03:35.322326 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 15:03:35.322392 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":159,"threads_started":1}
I20260928 15:03:35.323220 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d7743ea0a4cc8a75eb0998a489747" candidate_uuid: "854388a209b14bbaa18f8852a93f97c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3099b3cdb6034405a591329df5ddbdb4" is_pre_election: true
I20260928 15:03:35.323289 4103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d7743ea0a4cc8a75eb0998a489747" candidate_uuid: "854388a209b14bbaa18f8852a93f97c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" is_pre_election: true
I20260928 15:03:35.323421 4103 raft_consensus.cc:2379] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 854388a209b14bbaa18f8852a93f97c4 in current term 1: Already voted for candidate 3099b3cdb6034405a591329df5ddbdb4 in this term.
I20260928 15:03:35.323450 3954 catalog_manager.cc:5495] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3099b3cdb6034405a591329df5ddbdb4 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "3099b3cdb6034405a591329df5ddbdb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3099b3cdb6034405a591329df5ddbdb4" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36777 } health_report { overall_health: HEALTHY } } }
I20260928 15:03:35.323698 4173 leader_election.cc:302] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854388a209b14bbaa18f8852a93f97c4; no voters: 2f8d90f5854f4d9fa27ae9034db1a8f9, 3099b3cdb6034405a591329df5ddbdb4
I20260928 15:03:35.323854 4447 raft_consensus.cc:2726] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:03:35.342403 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.339473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.25.123.3:36777)
0928 15:03:35.339583 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 15:03:35.339585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:03:35.339594 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:03:35.339639 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:03:35.339640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:03:35.339665 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:03:35.339740 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:03:35.339958 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.340563 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.341388 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:03:35.341582 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:03:35.341646 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:03:35.341849 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:03:35.341980 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:03:35.342060 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:03:35.342175 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 15:03:35.342247 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:03:35.344446 4239 raft_consensus.cc:1261] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 1 FOLLOWER]: Refusing update from remote peer 3099b3cdb6034405a591329df5ddbdb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:35.344447 4103 raft_consensus.cc:1261] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 1 FOLLOWER]: Refusing update from remote peer 3099b3cdb6034405a591329df5ddbdb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:03:35.344774 4448 consensus_queue.cc:1035] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8d90f5854f4d9fa27ae9034db1a8f9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 34727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:35.344939 4451 consensus_queue.cc:1035] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854388a209b14bbaa18f8852a93f97c4" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 46727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:03:35.347591 4454 mvcc.cc:205] Tried to move back new op lower bound from 7334329611647705088 to 7334329611559165952. Current Snapshot: MvccSnapshot[applied={T|T < 7334329611647705088}]
I20260928 15:03:35.348627 4456 mvcc.cc:205] Tried to move back new op lower bound from 7334329611647705088 to 7334329611559165952. Current Snapshot: MvccSnapshot[applied={T|T < 7334329611647705088 or (T in {7334329611647705088})}]
I20260928 15:03:35.350456 3955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:35.350572 3955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:03:35.351423 3955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4036c03eb1f4d1fabf9396f28f3b365: Sending DeleteTablet for 3 replicas of tablet ae2d7743ea0a4cc8a75eb0998a489747
I20260928 15:03:35.351786 4218 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2d7743ea0a4cc8a75eb0998a489747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:35 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:48526
I20260928 15:03:35.351912 4082 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2d7743ea0a4cc8a75eb0998a489747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:35 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36284
I20260928 15:03:35.352069 4463 tablet_replica.cc:330] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: stopping tablet replica
I20260928 15:03:35.352161 4464 tablet_replica.cc:330] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9: stopping tablet replica
I20260928 15:03:35.352159 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2d7743ea0a4cc8a75eb0998a489747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:03:35 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:39752
I20260928 15:03:35.352267 4464 raft_consensus.cc:2229] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:35.352293 4463 raft_consensus.cc:2229] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:03:35.352421 4464 raft_consensus.cc:2258] T ae2d7743ea0a4cc8a75eb0998a489747 P 2f8d90f5854f4d9fa27ae9034db1a8f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:35.352543 4463 raft_consensus.cc:2258] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:35.352603 4465 tablet_replica.cc:330] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: stopping tablet replica
I20260928 15:03:35.352715 4465 raft_consensus.cc:2229] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:03:35.352730 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 4017
I20260928 15:03:35.352824 4465 raft_consensus.cc:2258] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:03:35.354480 4463 ts_tablet_manager.cc:1905] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:03:35.356781 4463 ts_tablet_manager.cc:1918] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:35.356875 4463 log.cc:1196] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/ae2d7743ea0a4cc8a75eb0998a489747
I20260928 15:03:35.357147 4463 ts_tablet_manager.cc:1939] T ae2d7743ea0a4cc8a75eb0998a489747 P 854388a209b14bbaa18f8852a93f97c4: Deleting consensus metadata
I20260928 15:03:35.357679 3939 catalog_manager.cc:4841] TS 854388a209b14bbaa18f8852a93f97c4 (127.25.123.2:46727): tablet ae2d7743ea0a4cc8a75eb0998a489747 (table test-table [id=77d3b59a0cd94eae850303f715caa684]) successfully deleted
I20260928 15:03:35.361012 4465 ts_tablet_manager.cc:1905] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 15:03:35.361492 3942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:34727 (error 108)
I20260928 15:03:35.361725 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 4154
I20260928 15:03:35.364970 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:03:35.364756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:34727 (local address 127.0.0.1:36290)
0928 15:03:35.364864 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 15:03:35.364918 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:34727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 15:03:35.365129 3942 catalog_manager.cc:4584] TS 2f8d90f5854f4d9fa27ae9034db1a8f9 (127.25.123.1:34727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae2d7743ea0a4cc8a75eb0998a489747: Network error: Client connection negotiation failed: client connection to 127.25.123.1:34727: connect: Connection refused (error 111)
I20260928 15:03:35.365751 4465 ts_tablet_manager.cc:1918] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:03:35.365892 4465 log.cc:1196] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/ae2d7743ea0a4cc8a75eb0998a489747
I20260928 15:03:35.366204 4465 ts_tablet_manager.cc:1939] T ae2d7743ea0a4cc8a75eb0998a489747 P 3099b3cdb6034405a591329df5ddbdb4: Deleting consensus metadata
I20260928 15:03:35.366789 3941 catalog_manager.cc:4841] TS 3099b3cdb6034405a591329df5ddbdb4 (127.25.123.3:36777): tablet ae2d7743ea0a4cc8a75eb0998a489747 (table test-table [id=77d3b59a0cd94eae850303f715caa684]) successfully deleted
I20260928 15:03:35.367633 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 4290
I20260928 15:03:35.373804 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 3923
2026-09-28T15:03:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2392 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:03:35 dist-test-slave-qnz0 krb5kdc[4469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:03:35 dist-test-slave-qnz0 krb5kdc[4469](info): set up 2 sockets
Sep 28 15:03:35 dist-test-slave-qnz0 krb5kdc[4469](info): commencing operation
krb5kdc: starting...
W20260928 15:03:36.561455 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:03:36 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:03:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/rangeradmin_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/rangeradmin_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/rangerlookup_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/rangerlookup_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
I20260928 15:03:36.790409 26092 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGgw5f9/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/postgres -l logfile start
2026-09-28 15:03:41.434 UTC [4500] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:03:41.434 UTC [4500] LOG: listening on IPv4 address "127.25.123.20", port 47001
2026-09-28 15:03:41.442 UTC [4500] LOG: listening on Unix socket "/tmp/.s.PGSQL.47001"
2026-09-28 15:03:41.471 UTC [4503] LOG: database system was shut down at 2026-09-28 15:03:38 UTC
2026-09-28 15:03:41.481 UTC [4500] LOG: database system is ready to accept connections
I20260928 15:03:42.537357 26092 mini_postgres.cc:96] Postgres bound to 47001
2026-09-28 15:03:42.546 UTC [4511] FATAL: database "slave" does not exist
127.25.123.20:47001 - accepting connections
I20260928 15:03:42.547012 26092 mini_ranger.cc:161] Starting Ranger...
I20260928 15:03:42.566480 26092 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:03:43.531446 26092 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:03:43.531553 26092 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-admin
2026-09-28 15:03:43,923 [I] DB FLAVOR :POSTGRES
2026-09-28 15:03:43,923 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:03:43,923 [I] Checking connection..
2026-09-28 15:03:43,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:03:44,297 [I] Checking connection passed.
2026-09-28 15:03:44,297 [I] --------- Verifying version history table ---------
2026-09-28 15:03:44,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:03:44,600 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:03:44,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:03:44,919 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:03:44,920 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:03:44,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:03:45.276 UTC [4576] WARNING: there is no transaction in progress
2026-09-28 15:03:45,285 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:03:45,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:03:45,622 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:03:45,623 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:03:45,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:03:45,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:03:46,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qnz0.c.gcp-upstream.internal','N') ;"
2026-09-28 15:03:46,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:03:47,040 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:03:47,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:03:47,400 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:03:47,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:03:47,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:03:48,056 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:03:48,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:03:49.668 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.692 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.719 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.742 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.765 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.797 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.882 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.893 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.921 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.937 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.953 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.978 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:49.994 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:50.012 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:50.030 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:50.045 UTC [4685] WARNING: there is no transaction in progress
2026-09-28 15:03:51,258 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:03:51,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:03:51,584 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:03:51,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:03:51,994 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:03:51,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:03:52,356 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:03:52,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:03:52,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:03:52,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qnz0.c.gcp-upstream.internal';"
2026-09-28 15:03:53,260 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:03:53,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:03:53,631 [I] DB_PATCHES have already been applied
I20260928 15:03:53.637995 26092 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGgw5f9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:03:53.638082 26092 mini_ranger.cc:193] Using host: 127.25.123.20
I20260928 15:03:53.641944 26092 mini_ranger.cc:227] Ranger admin URL: http://127.25.123.20:36425
Sep 28, 2026 3:03:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:03:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:03:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:03:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:03:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:03:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:03:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:03:56.218077 26092 mini_ranger.cc:160] Time spent starting Ranger: real 13.671s user 0.004s sys 0.009s
Sep 28, 2026 3:04:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:04:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 3:04:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:04:25 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:04:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:04:27.226241 26092 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/rangerkms_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/rangerkms_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.123.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.123.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:04:27 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607867, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:04:27.353961 26092 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:04:27.366914 26092 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:04:28.580307 26092 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:04:28.580446 26092 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms
I20260928 15:04:28.580470 26092 mini_ranger_kms.cc:227] Using postgres at 127.25.123.20:47001
2026-09-28 15:04:29,063 [I] DB FLAVOR :POSTGRES
2026-09-28 15:04:29,064 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:04:29,064 [I] Checking connection
2026-09-28 15:04:29,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:04:29,410 [I] connection success
2026-09-28 15:04:29,410 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:04:29,410 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:04:29,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:04:29,754 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:04:29,754 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:04:29,755 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:04:29,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:47001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:04:30,111 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:04:30.812528 26092 mini_ranger_kms.cc:310] Created kms service
I20260928 15:04:31.104978 26092 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:04:31.168699 26092 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:04:31 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607867, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:04:31.734217 26092 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:04:31.734356 26092 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:04:31.734378 26092 mini_ranger_kms.cc:240] Using host: 127.25.123.20
I20260928 15:04:31.738587 26092 mini_ranger_kms.cc:276] Ranger KMS PID: 4870
I20260928 15:04:31.738754 26092 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.123.20:34841
Sep 28, 2026 3:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:04:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:04:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:04:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:04:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 15:04:34.197294 26092 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.843s user 0.007s sys 0.261s
I20260928 15:04:34.197424 26092 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:04:34.197511 26092 mini_ranger_kms.cc:401] 127.25.123.20:34841/kms/v1/keys
Sep 28 15:04:34 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.123.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 15:04:35 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.123.20@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
Sep 28 15:04:35 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.123.20@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
Sep 28 15:04:38 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:04:38.465504 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37793
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:37793
--ranger_config_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:34841/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:04:38.583993 4921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:38.584270 4921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:38.584296 4921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:38.585862 4921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:04:38.585938 4921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:38.585954 4921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:38.585968 4921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:04:38.585980 4921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:04:38.588925 4921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.123.20:34841/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:37793
--ranger_config_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:04:38.589713 4921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:38.590121 4921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:38.598055 4927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:38.597980 4929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:38.598064 4926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:38.598642 4921 server_base.cc:1034] running on GCE node
I20260928 15:04:38.599071 4921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:38.599781 4921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:38.601115 4921 hybrid_clock.cc:648] HybridClock initialized: now 1790607878601072 us; error 64 us; skew 500 ppm
Sep 28 15:04:38 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:38.612316 4921 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:04:38.613266 4921 webserver.cc:460] Webserver started at http://127.25.123.62:39815/ using document root <none> and password file <none>
I20260928 15:04:38.613623 4921 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:38.613694 4921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:38.614022 4921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:38 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607878, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:04:38.704231 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "ccf0aac18e634f5abd10267ee144103c"
format_stamp: "Formatted at 2026-09-28 15:04:38 on dist-test-slave-qnz0"
server_key: "cc77bc6ef2b431b48a030dee50cb4e1b"
server_key_iv: "e680c8163673146edb7c6218dd78c6cd"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:38.704779 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "ccf0aac18e634f5abd10267ee144103c"
format_stamp: "Formatted at 2026-09-28 15:04:38 on dist-test-slave-qnz0"
server_key: "cc77bc6ef2b431b48a030dee50cb4e1b"
server_key_iv: "e680c8163673146edb7c6218dd78c6cd"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:38.706372 4921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:04:38.725620 4936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:38.725865 4921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:04:38.725970 4921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "ccf0aac18e634f5abd10267ee144103c"
format_stamp: "Formatted at 2026-09-28 15:04:38 on dist-test-slave-qnz0"
server_key: "cc77bc6ef2b431b48a030dee50cb4e1b"
server_key_iv: "e680c8163673146edb7c6218dd78c6cd"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:38.726100 4921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:38.741344 4921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:38.745069 4921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:38.745229 4921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:38.750617 4921 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:37793
I20260928 15:04:38.750633 4988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:37793 every 8 connection(s)
I20260928 15:04:38.751681 4921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:04:38.752882 4989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:38.756263 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 4921
I20260928 15:04:38.756383 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:04:38.756840 26092 external_mini_cluster.cc:1428] Setting key e65d9644d89e1b9ea02927c47ae16431
I20260928 15:04:38.757583 4989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c: Bootstrap starting.
I20260928 15:04:38.759176 4989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:38.760344 4989 log.cc:824] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:38.761871 4989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c: No bootstrap required, opened a new log
I20260928 15:04:38.765100 4989 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0aac18e634f5abd10267ee144103c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37793 } }
I20260928 15:04:38.765291 4989 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:38.765316 4989 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0aac18e634f5abd10267ee144103c, State: Initialized, Role: FOLLOWER
I20260928 15:04:38.765473 4989 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0aac18e634f5abd10267ee144103c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37793 } }
I20260928 15:04:38.765594 4989 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:04:38.765748 4989 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:04:38.765828 4989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 0 FOLLOWER]: Advancing to term 1
Sep 28 15:04:38 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:04:38.767379 4989 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0aac18e634f5abd10267ee144103c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37793 } }
I20260928 15:04:38.767540 4989 leader_election.cc:302] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf0aac18e634f5abd10267ee144103c; no voters:
I20260928 15:04:38.767750 4989 leader_election.cc:288] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:04:38.768126 4995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:04:38.768414 4995 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [term 1 LEADER]: Becoming Leader. State: Replica: ccf0aac18e634f5abd10267ee144103c, State: Running, Role: LEADER
I20260928 15:04:38.768512 4995 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0aac18e634f5abd10267ee144103c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37793 } }
I20260928 15:04:38.768236 4989 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:04:38.768878 4994 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf0aac18e634f5abd10267ee144103c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0aac18e634f5abd10267ee144103c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37793 } } }
I20260928 15:04:38.768900 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf0aac18e634f5abd10267ee144103c. Latest consensus state: current_term: 1 leader_uuid: "ccf0aac18e634f5abd10267ee144103c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0aac18e634f5abd10267ee144103c" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37793 } } }
I20260928 15:04:38.769037 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [sys.catalog]: This master's current role is: LEADER
I20260928 15:04:38.769037 4994 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c [sys.catalog]: This master's current role is: LEADER
I20260928 15:04:38.769277 4999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:04:38.770128 4999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:04:38.770996 4989 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:04:38.777232 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:38.757481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.25.123.62:37793)
0928 15:04:38.757696 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:38.757708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:38.757745 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:04:38.758400 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:38.758410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:38.758469 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:38.758682 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:38.759385 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:38.761172 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:38.762607 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:38.763060 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:38.763161 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:38.766528 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:38.766535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:38.766535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:38.766551 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:38.770553 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:38.771135 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:38.771139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:38.771139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:38.771241 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:38.771484 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:38.771484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:38.771485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:38.773911 (+ 2426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:38.775263 (+ 1352us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:38.775529 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:04:38.776669 (+ 1140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260928 15:04:38.778332 4999 catalog_manager.cc:1347] Generated new cluster ID: 8e7b3c362db44e3c8c40ad2ca52f0a72
I20260928 15:04:38.778462 4999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:04:38.794010 4999 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:04:38.795102 4999 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:04:38.805787 4999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c: Generated new TSK 0
I20260928 15:04:38.806044 4999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:04:39.797883 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:34841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37793
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:04:39.906843 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:39.907218 5035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:39.907274 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:39.908957 5035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:39.909040 5035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:39.909098 5035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:04:39.911017 5035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.123.20:34841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:04:39.911566 5035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:39.911859 5035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:39.920426 5044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:39.920426 5042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:39.920496 5041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:39.921401 5035 server_base.cc:1034] running on GCE node
I20260928 15:04:39.921775 5035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:39.922109 5035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:39.925640 5035 hybrid_clock.cc:648] HybridClock initialized: now 1790607879925582 us; error 68 us; skew 500 ppm
Sep 28 15:04:39 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:39.931083 5035 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:04:39.931882 5035 webserver.cc:460] Webserver started at http://127.25.123.1:44163/ using document root <none> and password file <none>
I20260928 15:04:39.932229 5035 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:39.932322 5035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:39.932516 5035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:39 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:04:39.953611 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "50617c3e3f8f441cb24891768683f4fd"
format_stamp: "Formatted at 2026-09-28 15:04:39 on dist-test-slave-qnz0"
server_key: "05ea39a205e1bc324b5c02db88ea2d86"
server_key_iv: "4062822ab3903cf041c9accd8483e113"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:39.954241 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "50617c3e3f8f441cb24891768683f4fd"
format_stamp: "Formatted at 2026-09-28 15:04:39 on dist-test-slave-qnz0"
server_key: "05ea39a205e1bc324b5c02db88ea2d86"
server_key_iv: "4062822ab3903cf041c9accd8483e113"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:39.958904 5035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 15:04:39.980639 5051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:39.981405 5035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:04:39.981572 5035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "50617c3e3f8f441cb24891768683f4fd"
format_stamp: "Formatted at 2026-09-28 15:04:39 on dist-test-slave-qnz0"
server_key: "05ea39a205e1bc324b5c02db88ea2d86"
server_key_iv: "4062822ab3903cf041c9accd8483e113"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:39.981726 5035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:39.998207 5035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:40.007318 5035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:40.007604 5035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:40.007936 5035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:04:40.008438 5035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:04:40.008534 5035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.008621 5035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:04:40.008661 5035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.022042 5035 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:42361
I20260928 15:04:40.022569 5166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:42361 every 8 connection(s)
I20260928 15:04:40.023319 5035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:04:40.027815 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 5035
I20260928 15:04:40.027954 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:04:40.028306 26092 external_mini_cluster.cc:1428] Setting key 2fc013882fcb9618617628f1a2c007ac
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607879, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:04:40.042718 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.025404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:40015 (local address 127.25.123.62:37793)
0928 15:04:40.025576 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.025578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.025801 (+ 223us) server_negotiation.cc:407] Connection header received
0928 15:04:40.026512 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.026514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.026530 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.026622 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:40.029688 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.030941 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.033284 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.033681 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.033749 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.037194 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:40.037196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:40.037196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:40.037206 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:40.039259 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.040605 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.040607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.040608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.040709 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.041131 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.041132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.041133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.041571 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:40.041693 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.042388 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.042563 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 15:04:40.043963 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.024978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37793 (local address 127.25.123.1:40015)
0928 15:04:40.025620 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.025677 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.026378 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.027302 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.027316 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.028066 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:04:40.029034 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.029249 (+ 215us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.031128 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.031130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.033105 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.033111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.033878 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.033879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.034066 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.034922 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:04:40.034935 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:04:40.037030 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:04:40.039398 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:40.039413 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:40.039417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:40.040276 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:40.040803 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:40.040804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:40.040805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:40.041036 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:40.041769 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:04:40.041784 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:04:40.042150 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.043299 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.043592 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":218,"threads_started":1}
I20260928 15:04:40.046168 5167 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37793
I20260928 15:04:40.046329 5167 heartbeater.cc:464] Registering TS with master...
I20260928 15:04:40.046778 5167 heartbeater.cc:511] Master 127.25.123.62:37793 requested a full tablet report, sending...
I20260928 15:04:40.047881 4953 ts_manager.cc:194] Registered new tserver with Master: 50617c3e3f8f441cb24891768683f4fd (127.25.123.1:42361)
I20260928 15:04:40.049196 4953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:40015
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:04:40.082157 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:34841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37793
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:04:40.228096 5173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:40.228454 5173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:40.228648 5173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:40.230494 5173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:40.230599 5173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:40.230654 5173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:04:40.232820 5173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.123.20:34841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:04:40.233151 5173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:40.233466 5173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:40.239740 5184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:40.239984 5173 server_base.cc:1034] running on GCE node
W20260928 15:04:40.240149 5181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:40.240260 5180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:40.240550 5173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:40.240927 5173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:40.242112 5173 hybrid_clock.cc:648] HybridClock initialized: now 1790607880242093 us; error 37 us; skew 500 ppm
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:40.245187 5173 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:04:40.245728 5173 webserver.cc:460] Webserver started at http://127.25.123.2:35707/ using document root <none> and password file <none>
I20260928 15:04:40.246019 5173 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:40.246100 5173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:40.246279 5173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:04:40.262198 5173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "b09c8ba6ee374166a14c7f6398b0b675"
format_stamp: "Formatted at 2026-09-28 15:04:40 on dist-test-slave-qnz0"
server_key: "75e28ee76280d19da5bd51c749bc4f0e"
server_key_iv: "0653892c0168e023b01ca1fb5084215e"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:40.262846 5173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "b09c8ba6ee374166a14c7f6398b0b675"
format_stamp: "Formatted at 2026-09-28 15:04:40 on dist-test-slave-qnz0"
server_key: "75e28ee76280d19da5bd51c749bc4f0e"
server_key_iv: "0653892c0168e023b01ca1fb5084215e"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:40.265044 5173 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:40.299038 5191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.299364 5173 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260928 15:04:40.299480 5173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "b09c8ba6ee374166a14c7f6398b0b675"
format_stamp: "Formatted at 2026-09-28 15:04:40 on dist-test-slave-qnz0"
server_key: "75e28ee76280d19da5bd51c749bc4f0e"
server_key_iv: "0653892c0168e023b01ca1fb5084215e"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:40.299613 5173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:40.347393 5173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:40.349936 5173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:40.350121 5173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:40.350353 5173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:04:40.350752 5173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:04:40.350802 5173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.350844 5173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:04:40.350867 5173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.357963 5173 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:43059
I20260928 15:04:40.358434 5173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:04:40.361714 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 5173
I20260928 15:04:40.361841 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:04:40.362046 26092 external_mini_cluster.cc:1428] Setting key 5fc8a4cd48aafbb78f977bed63966524
I20260928 15:04:40.360713 5306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:43059 every 8 connection(s)
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:04:40.385068 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.361514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:51989 (local address 127.25.123.62:37793)
0928 15:04:40.361566 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.361568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.372774 (+ 11206us) server_negotiation.cc:407] Connection header received
0928 15:04:40.373618 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.373620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.373643 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.373735 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:40.374791 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.375699 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.377552 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.377941 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.377992 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.381259 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:40.381261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:40.381262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:40.381271 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:40.383147 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.383767 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.383768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.383769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.383834 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.384136 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.384137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.384137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.384306 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:40.384363 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.384833 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.384950 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:04:40.386153 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.361186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37793 (local address 127.25.123.2:51989)
0928 15:04:40.372531 (+ 11345us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.372620 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.373454 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.373912 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.373919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.374298 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:04:40.374637 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.374644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.375850 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.375852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.377421 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.377424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.378106 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.378108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.378293 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.379130 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:04:40.379143 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:04:40.381098 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:04:40.383291 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:40.383301 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:40.383305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:40.383667 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:40.383907 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:40.383908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:40.383908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:40.384060 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:40.384427 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:04:40.384435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:04:40.384638 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.385639 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.385854 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11268,"thread_start_us":114,"threads_started":1}
I20260928 15:04:40.386953 5307 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37793
I20260928 15:04:40.387243 5307 heartbeater.cc:464] Registering TS with master...
I20260928 15:04:40.387553 5307 heartbeater.cc:511] Master 127.25.123.62:37793 requested a full tablet report, sending...
I20260928 15:04:40.388084 4953 ts_manager.cc:194] Registered new tserver with Master: b09c8ba6ee374166a14c7f6398b0b675 (127.25.123.2:43059)
I20260928 15:04:40.388690 4953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:51989
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:04:40.426772 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:34841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37793
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:04:40.570991 5313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:04:40.571455 5313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:04:40.571568 5313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:04:40.573945 5313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:04:40.574121 5313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:04:40.574242 5313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:04:40.577234 5313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.123.20:34841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:04:40.577857 5313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:04:40.578286 5313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:04:40.585551 5318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:40.585564 5321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:04:40.586071 5319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:04:40.586668 5313 server_base.cc:1034] running on GCE node
I20260928 15:04:40.587435 5313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:04:40.587792 5313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:04:40.589021 5313 hybrid_clock.cc:648] HybridClock initialized: now 1790607880588972 us; error 50 us; skew 500 ppm
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:04:40.592933 5313 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:04:40.593677 5313 webserver.cc:460] Webserver started at http://127.25.123.3:36107/ using document root <none> and password file <none>
I20260928 15:04:40.593995 5313 fs_manager.cc:359] Metadata directory not provided
I20260928 15:04:40.594069 5313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:04:40.594221 5313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:04:40.616080 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "5c7e0f197f4a4ad890928dea93ec80a0"
format_stamp: "Formatted at 2026-09-28 15:04:40 on dist-test-slave-qnz0"
server_key: "4c8392dc1a7a041d92d350c104abd6aa"
server_key_iv: "dd629c825c7e82d89d4f09f2e0d7ee07"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:40.616618 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7e0f197f4a4ad890928dea93ec80a0"
format_stamp: "Formatted at 2026-09-28 15:04:40 on dist-test-slave-qnz0"
server_key: "4c8392dc1a7a041d92d350c104abd6aa"
server_key_iv: "dd629c825c7e82d89d4f09f2e0d7ee07"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:40.618642 5313 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:04:40.638540 5328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.638782 5313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:04:40.638892 5313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "5c7e0f197f4a4ad890928dea93ec80a0"
format_stamp: "Formatted at 2026-09-28 15:04:40 on dist-test-slave-qnz0"
server_key: "4c8392dc1a7a041d92d350c104abd6aa"
server_key_iv: "dd629c825c7e82d89d4f09f2e0d7ee07"
server_key_version: "kuduclusterkey@0"
I20260928 15:04:40.639019 5313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:04:40.664678 5313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:04:40.668207 5313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:04:40.668429 5313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:04:40.668720 5313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:04:40.669133 5313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:04:40.669188 5313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.669229 5313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:04:40.669256 5313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:04:40.677253 5313 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:36617
I20260928 15:04:40.677683 5442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:36617 every 8 connection(s)
I20260928 15:04:40.677779 5313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:04:40.686686 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 5313
I20260928 15:04:40.686803 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:04:40.687026 26092 external_mini_cluster.cc:1428] Setting key 66a9b8f630502e37b8f97aeb2e81fc80
I20260928 15:04:40.690464 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.678646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:48603 (local address 127.25.123.62:37793)
0928 15:04:40.678703 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.678705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.678972 (+ 267us) server_negotiation.cc:407] Connection header received
0928 15:04:40.679559 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.679561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.679574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.679641 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:40.680648 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.681262 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.682434 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.682769 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.683104 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.685763 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:40.685765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:40.685765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:40.685783 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:40.687782 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.689338 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.689339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.689340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.689413 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.689720 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.689721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.689721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.689852 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:40.689928 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.690267 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.690366 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:04:40.690693 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.678611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37793 (local address 127.25.123.3:48603)
0928 15:04:40.678792 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.678838 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.679442 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.679743 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.679752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.679967 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:04:40.680362 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.680376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.681390 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.681392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.682332 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.682335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.683208 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.683210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.683373 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.684086 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:04:40.684097 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:04:40.685627 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:04:40.688828 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:40.688843 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:40.688846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:40.689243 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:40.689495 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:04:40.689495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:04:40.689496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:04:40.689631 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:04:40.689930 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:04:40.689937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:04:40.690103 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.690221 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.690401 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260928 15:04:40.691344 5443 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37793
I20260928 15:04:40.691478 5443 heartbeater.cc:464] Registering TS with master...
I20260928 15:04:40.691686 5443 heartbeater.cc:511] Master 127.25.123.62:37793 requested a full tablet report, sending...
I20260928 15:04:40.692126 4953 ts_manager.cc:194] Registered new tserver with Master: 5c7e0f197f4a4ad890928dea93ec80a0 (127.25.123.3:36617)
I20260928 15:04:40.692615 4953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:48603
I20260928 15:04:40.700182 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:04:40.714164 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.701906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.25.123.62:37793)
0928 15:04:40.701956 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.701958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.702608 (+ 650us) server_negotiation.cc:407] Connection header received
0928 15:04:40.702692 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.702693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.702705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.702756 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:04:40.703922 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.705581 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.707559 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.707881 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.707931 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.710407 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:04:40.710410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:04:40.710410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:04:40.710421 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:04:40.712254 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.712769 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.712771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.712771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.712850 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:04:40.713045 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:04:40.713046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:04:40.713046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:04:40.713190 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:04:40.713287 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.714010 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.714089 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:04:40.716293 4953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:04:40.718750 4953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:04:40.735554 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.725488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36617 (local address 127.0.0.1:51668)
0928 15:04:40.730336 (+ 4848us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.730347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.730402 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.730707 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.730711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.730713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:40.730801 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.730803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.732521 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.732521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.733749 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.733752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.735160 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.735161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.735279 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.735296 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.735378 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.735445 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1518,"spinlock_wait_cycles":1408,"thread_start_us":640,"threads_started":1}
I20260928 15:04:40.736793 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.725813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.25.123.3:36617)
0928 15:04:40.730306 (+ 4493us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.730313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.730453 (+ 140us) server_negotiation.cc:407] Connection header received
0928 15:04:40.730499 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.730502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.730520 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.730591 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:40.730937 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.732359 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.733880 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.734951 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.735027 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.735120 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.736200 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.736253 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4444,"thread_start_us":90,"threads_started":1}
I20260928 15:04:40.737795 5364 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 (DEFAULT_TABLE table=test-table [id=633719c2127d470081386a606d3cd045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:04:40.738189 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:40.743083 5461 tablet_bootstrap.cc:492] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Bootstrap starting.
I20260928 15:04:40.744253 5461 tablet_bootstrap.cc:654] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:40.744686 5461 log.cc:824] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:40.745390 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.725490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:42361 (local address 127.0.0.1:48322)
0928 15:04:40.729915 (+ 4425us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.729988 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.730082 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.734950 (+ 4868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.734951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.734953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:40.734995 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.734996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.742473 (+ 7477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.742475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.743668 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.743674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.744976 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.744978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.745055 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.745061 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.745223 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.745298 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4366,"mutex_wait_us":5,"thread_start_us":636,"threads_started":1}
I20260928 15:04:40.745630 5461 tablet_bootstrap.cc:492] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: No bootstrap required, opened a new log
I20260928 15:04:40.745713 5461 ts_tablet_manager.cc:1397] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:04:40.745769 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.725811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.25.123.1:42361)
0928 15:04:40.733934 (+ 8123us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.733941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.733954 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:04:40.733993 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.733995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.734015 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.735079 (+ 1064us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:40.735160 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.742346 (+ 7186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.743799 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.744743 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.744830 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.745606 (+ 776us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.745639 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.745692 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8076,"thread_start_us":95,"threads_started":1}
I20260928 15:04:40.746531 5090 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 (DEFAULT_TABLE table=test-table [id=633719c2127d470081386a606d3cd045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:04:40.746918 5090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:40.748039 5461 raft_consensus.cc:348] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.748225 5461 raft_consensus.cc:374] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:40.748263 5461 raft_consensus.cc:729] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7e0f197f4a4ad890928dea93ec80a0, State: Initialized, Role: FOLLOWER
I20260928 15:04:40.748420 5461 consensus_queue.cc:260] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.748761 5461 ts_tablet_manager.cc:1428] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 15:04:40.748916 5443 heartbeater.cc:503] Master 127.25.123.62:37793 was elected leader, sending a full tablet report...
I20260928 15:04:40.754654 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.726437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43059 (local address 127.0.0.1:34936)
0928 15:04:40.734140 (+ 7703us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.734151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.734192 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.734767 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.734768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.734771 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:40.734831 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.734833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.750874 (+ 16041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.750876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.752318 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.752323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.754317 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.754318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.754404 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.754409 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.754483 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.754548 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7686,"thread_start_us":82,"threads_started":1}
I20260928 15:04:40.754832 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.726560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.25.123.2:43059)
0928 15:04:40.734451 (+ 7891us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.734456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.734470 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:04:40.734515 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.734517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.734540 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.734647 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:40.736239 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.742841 (+ 6602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.752548 (+ 9707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.754020 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.754107 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.754223 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.754612 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.754696 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7822,"thread_start_us":70,"threads_started":1}
I20260928 15:04:40.755438 5464 tablet_bootstrap.cc:492] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd: Bootstrap starting.
I20260928 15:04:40.756182 5221 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 (DEFAULT_TABLE table=test-table [id=633719c2127d470081386a606d3cd045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:04:40.756579 5464 tablet_bootstrap.cc:654] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:40.756841 5221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:04:40.757046 5464 log.cc:824] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:40.758142 5464 tablet_bootstrap.cc:492] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd: No bootstrap required, opened a new log
I20260928 15:04:40.758230 5464 ts_tablet_manager.cc:1397] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:04:40.761657 5464 raft_consensus.cc:348] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.761870 5464 raft_consensus.cc:374] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:40.761912 5464 raft_consensus.cc:729] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50617c3e3f8f441cb24891768683f4fd, State: Initialized, Role: FOLLOWER
I20260928 15:04:40.762310 5464 consensus_queue.cc:260] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.762827 5464 ts_tablet_manager.cc:1428] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 15:04:40.763036 5167 heartbeater.cc:503] Master 127.25.123.62:37793 was elected leader, sending a full tablet report...
I20260928 15:04:40.763836 5467 tablet_bootstrap.cc:492] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Bootstrap starting.
I20260928 15:04:40.765057 5467 tablet_bootstrap.cc:654] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Neither blocks nor log segments found. Creating new log.
I20260928 15:04:40.765458 5467 log.cc:824] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Log is configured to *not* fsync() on all Append() calls
I20260928 15:04:40.766512 5467 tablet_bootstrap.cc:492] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: No bootstrap required, opened a new log
I20260928 15:04:40.766606 5467 ts_tablet_manager.cc:1397] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:04:40.769075 5467 raft_consensus.cc:348] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.769351 5467 raft_consensus.cc:374] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:04:40.769397 5467 raft_consensus.cc:729] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09c8ba6ee374166a14c7f6398b0b675, State: Initialized, Role: FOLLOWER
I20260928 15:04:40.769574 5467 consensus_queue.cc:260] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.769927 5467 ts_tablet_manager.cc:1428] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:04:40.769980 5307 heartbeater.cc:503] Master 127.25.123.62:37793 was elected leader, sending a full tablet report...
W20260928 15:04:40.774721 5168 tablet.cc:2369] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:04:40.860579 5308 tablet.cc:2369] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:04:40 dist-test-slave-qnz0 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607880, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:04:40.914996 5466 raft_consensus.cc:482] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:04:40.915184 5466 raft_consensus.cc:504] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.915935 5466 leader_election.cc:288] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09c8ba6ee374166a14c7f6398b0b675 (127.25.123.2:43059), 5c7e0f197f4a4ad890928dea93ec80a0 (127.25.123.3:36617)
I20260928 15:04:40.921160 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.916412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:58195 (local address 127.25.123.2:43059)
0928 15:04:40.916464 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.916467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.917062 (+ 595us) server_negotiation.cc:407] Connection header received
0928 15:04:40.917118 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.917119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.917134 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.917216 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:40.917445 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.918201 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.919976 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.920635 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.920698 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.920711 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.920995 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.921045 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:04:40.921520 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.916263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43059 (local address 127.25.123.1:58195)
0928 15:04:40.916767 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.916782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.916859 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.917242 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.917243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.917245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:40.917355 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.917358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.918300 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.918302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.919829 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.919834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.920799 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.920800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.920898 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.920902 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.921334 (+ 432us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.921399 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":113,"threads_started":1}
I20260928 15:04:40.922317 5260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea77b34d0c42b39dcb1b1c2c9d1df8" candidate_uuid: "50617c3e3f8f441cb24891768683f4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09c8ba6ee374166a14c7f6398b0b675" is_pre_election: true
I20260928 15:04:40.922562 5260 raft_consensus.cc:2454] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50617c3e3f8f441cb24891768683f4fd in term 0.
I20260928 15:04:40.922907 5054 leader_election.cc:302] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50617c3e3f8f441cb24891768683f4fd, b09c8ba6ee374166a14c7f6398b0b675; no voters:
I20260928 15:04:40.923090 5466 raft_consensus.cc:2781] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:04:40.923151 5466 raft_consensus.cc:482] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:04:40.923187 5466 raft_consensus.cc:3037] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:04:40.924610 5466 raft_consensus.cc:504] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.924816 5466 leader_election.cc:288] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [CANDIDATE]: Term 1 election: Requested vote from peers b09c8ba6ee374166a14c7f6398b0b675 (127.25.123.2:43059), 5c7e0f197f4a4ad890928dea93ec80a0 (127.25.123.3:36617)
I20260928 15:04:40.925098 5260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea77b34d0c42b39dcb1b1c2c9d1df8" candidate_uuid: "50617c3e3f8f441cb24891768683f4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09c8ba6ee374166a14c7f6398b0b675"
I20260928 15:04:40.925207 5260 raft_consensus.cc:3037] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:04:40.926373 5260 raft_consensus.cc:2454] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50617c3e3f8f441cb24891768683f4fd in term 1.
I20260928 15:04:40.926609 5054 leader_election.cc:302] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50617c3e3f8f441cb24891768683f4fd, b09c8ba6ee374166a14c7f6398b0b675; no voters:
I20260928 15:04:40.926730 5466 raft_consensus.cc:2781] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:04:40.926925 5466 raft_consensus.cc:686] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [term 1 LEADER]: Becoming Leader. State: Replica: 50617c3e3f8f441cb24891768683f4fd, State: Running, Role: LEADER
I20260928 15:04:40.927044 5466 consensus_queue.cc:237] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } }
I20260928 15:04:40.928071 4951 catalog_manager.cc:5495] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 50617c3e3f8f441cb24891768683f4fd (127.25.123.1). New cstate: current_term: 1 leader_uuid: "50617c3e3f8f441cb24891768683f4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50617c3e3f8f441cb24891768683f4fd" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 } health_report { overall_health: UNKNOWN } } }
W20260928 15:04:40.928542 5444 tablet.cc:2369] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:04:40.932314 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.916584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36617 (local address 127.25.123.1:36855)
0928 15:04:40.927578 (+ 10994us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:40.927595 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:40.927717 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:40.927937 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:04:40.927937 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:04:40.927940 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:04:40.928017 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.928019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.929732 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.929844 (+ 112us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.931078 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:04:40.931083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.931946 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.931947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:04:40.932036 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.932039 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:04:40.932120 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:04:40.932186 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10964,"thread_start_us":73,"threads_started":1}
I20260928 15:04:40.932472 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.916634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:36855 (local address 127.25.123.3:36617)
0928 15:04:40.916685 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.916687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.927679 (+ 10992us) server_negotiation.cc:407] Connection header received
0928 15:04:40.927811 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.927812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.927824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.927856 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:04:40.928102 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.929514 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.931214 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.931807 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.931863 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.931874 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.932143 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.932232 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 15:04:40.933154 5397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea77b34d0c42b39dcb1b1c2c9d1df8" candidate_uuid: "50617c3e3f8f441cb24891768683f4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" is_pre_election: true
I20260928 15:04:40.933568 5397 raft_consensus.cc:2454] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50617c3e3f8f441cb24891768683f4fd in term 0.
I20260928 15:04:40.933754 5396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea77b34d0c42b39dcb1b1c2c9d1df8" candidate_uuid: "50617c3e3f8f441cb24891768683f4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7e0f197f4a4ad890928dea93ec80a0"
I20260928 15:04:40.933840 5396 raft_consensus.cc:3037] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:04:40.936743 5396 raft_consensus.cc:2454] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50617c3e3f8f441cb24891768683f4fd in term 1.
I20260928 15:04:40.961057 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:04:40.955765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.25.123.1:42361)
0928 15:04:40.955835 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:04:40.955837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:04:40.955846 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:04:40.955905 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:04:40.955906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:04:40.955932 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:04:40.956011 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:04:40.957283 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.958234 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.959484 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:04:40.959791 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:04:40.959854 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:04:40.960066 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:04:40.960375 (+ 309us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:04:40.960505 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 15:04:40.960915 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 15:04:40.960944 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:04:40.972954 5260 raft_consensus.cc:1261] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 1 FOLLOWER]: Refusing update from remote peer 50617c3e3f8f441cb24891768683f4fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:04:40.972954 5396 raft_consensus.cc:1261] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 1 FOLLOWER]: Refusing update from remote peer 50617c3e3f8f441cb24891768683f4fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:04:40.973402 5466 consensus_queue.cc:1035] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09c8ba6ee374166a14c7f6398b0b675" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:04:40.973555 5472 consensus_queue.cc:1035] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 50617c3e3f8f441cb24891768683f4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7e0f197f4a4ad890928dea93ec80a0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:04:40.980656 4953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:04:40.981006 4953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:04:40.981916 5476 mvcc.cc:205] Tried to move back new op lower bound from 7334329880451133440 to 7334329880277852160. Current Snapshot: MvccSnapshot[applied={T|T < 7334329880451133440}]
I20260928 15:04:40.983474 4953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccf0aac18e634f5abd10267ee144103c: Sending DeleteTablet for 3 replicas of tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8
I20260928 15:04:40.983944 5364 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:04:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:51668
I20260928 15:04:40.984436 5090 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:04:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260928 15:04:40.984501 5484 tablet_replica.cc:330] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: stopping tablet replica
I20260928 15:04:40.984731 5484 raft_consensus.cc:2229] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:04:40.984923 5484 raft_consensus.cc:2258] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:04:40.985482 5221 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:04:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:34936
I20260928 15:04:40.985783 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 5035
I20260928 15:04:40.985805 5486 tablet_replica.cc:330] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: stopping tablet replica
I20260928 15:04:40.985961 5486 raft_consensus.cc:2229] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:04:40.986101 5486 raft_consensus.cc:2258] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:04:40.993017 5486 ts_tablet_manager.cc:1905] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:04:40.995332 5486 ts_tablet_manager.cc:1918] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:04:40.995460 5486 log.cc:1196] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/e2ea77b34d0c42b39dcb1b1c2c9d1df8
I20260928 15:04:40.995780 5486 ts_tablet_manager.cc:1939] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P b09c8ba6ee374166a14c7f6398b0b675: Deleting consensus metadata
I20260928 15:04:40.997068 4939 catalog_manager.cc:4841] TS b09c8ba6ee374166a14c7f6398b0b675 (127.25.123.2:43059): tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 (table test-table [id=633719c2127d470081386a606d3cd045]) successfully deleted
I20260928 15:04:40.997831 5484 ts_tablet_manager.cc:1905] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:04:41.001406 5484 ts_tablet_manager.cc:1918] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:04:41.001546 5484 log.cc:1196] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/e2ea77b34d0c42b39dcb1b1c2c9d1df8
I20260928 15:04:41.002277 5484 ts_tablet_manager.cc:1939] T e2ea77b34d0c42b39dcb1b1c2c9d1df8 P 5c7e0f197f4a4ad890928dea93ec80a0: Deleting consensus metadata
I20260928 15:04:41.002882 4937 catalog_manager.cc:4841] TS 5c7e0f197f4a4ad890928dea93ec80a0 (127.25.123.3:36617): tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8 (table test-table [id=633719c2127d470081386a606d3cd045]) successfully deleted
W20260928 15:04:41.005240 4938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:42361 (error 108)
W20260928 15:04:41.006476 5458 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:04:41.006143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:42361 (local address 127.0.0.1:48350)
0928 15:04:41.006225 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 15:04:41.006245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:04:41.006329 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:04:41.006405 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:42361: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:04:41.006660 4938 catalog_manager.cc:4584] TS 50617c3e3f8f441cb24891768683f4fd (127.25.123.1:42361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2ea77b34d0c42b39dcb1b1c2c9d1df8: Network error: Client connection negotiation failed: client connection to 127.25.123.1:42361: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 15:04:41.006852 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 5173
I20260928 15:04:41.015843 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 5313
I20260928 15:04:41.023104 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 4921
I20260928 15:04:41.036201 26092 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:04:41.382295 26092 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:04:41.382556 26092 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:04:41.728825 26092 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:04:41.729 UTC [4500] LOG: received smart shutdown request
2026-09-28 15:04:41.734 UTC [4500] LOG: background worker "logical replication launcher" (PID 4509) exited with exit code 1
2026-09-28 15:04:41.735 UTC [4504] LOG: shutting down
2026-09-28 15:04:41.991 UTC [4500] LOG: database system is shut down
2026-09-28T15:04:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (66752 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:04:42 dist-test-slave-qnz0 krb5kdc[5497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:04:42 dist-test-slave-qnz0 krb5kdc[5497](info): set up 2 sockets
Sep 28 15:04:42 dist-test-slave-qnz0 krb5kdc[5497](info): commencing operation
krb5kdc: starting...
W20260928 15:04:43.251365 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:04:43 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:04:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/rangeradmin_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/rangeradmin_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/rangerlookup_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/rangerlookup_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
I20260928 15:04:43.478448 26092 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGgw5f9/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/postgres -l logfile start
2026-09-28 15:04:47.238 UTC [5527] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:04:47.238 UTC [5527] LOG: listening on IPv4 address "127.25.123.20", port 53055
2026-09-28 15:04:47.244 UTC [5527] LOG: listening on Unix socket "/tmp/.s.PGSQL.53055"
2026-09-28 15:04:47.285 UTC [5530] LOG: database system was shut down at 2026-09-28 15:04:44 UTC
2026-09-28 15:04:47.291 UTC [5527] LOG: database system is ready to accept connections
I20260928 15:04:48.312016 26092 mini_postgres.cc:96] Postgres bound to 53055
2026-09-28 15:04:48.318 UTC [5538] FATAL: database "slave" does not exist
127.25.123.20:53055 - accepting connections
I20260928 15:04:48.319597 26092 mini_ranger.cc:161] Starting Ranger...
I20260928 15:04:48.338397 26092 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:04:49.279750 26092 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:04:49.279903 26092 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-admin
2026-09-28 15:04:49,605 [I] DB FLAVOR :POSTGRES
2026-09-28 15:04:49,605 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:04:49,606 [I] Checking connection..
2026-09-28 15:04:49,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:04:49,930 [I] Checking connection passed.
2026-09-28 15:04:49,930 [I] --------- Verifying version history table ---------
2026-09-28 15:04:49,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:50,258 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:04:50,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:50,538 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:04:50,538 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:04:50,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:04:50.841 UTC [5602] WARNING: there is no transaction in progress
2026-09-28 15:04:50,849 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:04:50,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:04:51,120 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:04:51,121 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:04:51,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:04:51,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:04:51,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qnz0.c.gcp-upstream.internal','N') ;"
2026-09-28 15:04:51,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:04:52,225 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:04:52,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:04:52,573 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:04:52,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:04:52,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:04:53,174 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:04:53,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:04:54.487 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.511 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.532 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.552 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.573 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.594 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.665 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.674 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.693 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.707 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.720 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.738 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.753 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.764 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.775 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:54.786 UTC [5711] WARNING: there is no transaction in progress
2026-09-28 15:04:55,916 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:04:55,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:04:56,235 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:04:56,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:04:56,552 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:04:56,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:04:56,848 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:04:56,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:04:57,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:04:57,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qnz0.c.gcp-upstream.internal';"
2026-09-28 15:04:57,687 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:04:57,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:04:57,987 [I] DB_PATCHES have already been applied
I20260928 15:04:57.994093 26092 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGgw5f9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:04:57.994184 26092 mini_ranger.cc:193] Using host: 127.25.123.20
I20260928 15:04:57.999039 26092 mini_ranger.cc:227] Ranger admin URL: http://127.25.123.20:50755
Sep 28, 2026 3:04:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:04:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:04:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:04:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:04:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:04:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:04:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:04:59.217463 26092 mini_ranger.cc:160] Time spent starting Ranger: real 10.898s user 0.000s sys 0.011s
Sep 28, 2026 3:05:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:05:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:05:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:05:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:05:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:05:25.808640 26092 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.123.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.123.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/rangerkms_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.123.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/rangerkms_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.123.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.123.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.123.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/HTTP_127.25.123.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:05:25 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607925, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:05:25.925271 26092 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:05:25.938565 26092 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:05:27.372237 26092 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:05:27.372342 26092 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms
I20260928 15:05:27.372362 26092 mini_ranger_kms.cc:227] Using postgres at 127.25.123.20:53055
2026-09-28 15:05:27,576 [I] DB FLAVOR :POSTGRES
2026-09-28 15:05:27,577 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:05:27,577 [I] Checking connection
2026-09-28 15:05:27,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:05:27,884 [I] connection success
2026-09-28 15:05:27,884 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:05:27,885 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:05:27,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:05:28,190 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:05:28,191 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:05:28,191 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:05:28,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.123.20:53055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:05:28,544 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:05:29.241477 26092 mini_ranger_kms.cc:310] Created kms service
I20260928 15:05:29.465103 26092 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:05:29.521261 26092 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:05:29 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607925, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:30.060796 26092 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:05:30.061030 26092 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGgw5f9/build/release/bin/postgresql.jar:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGgw5f9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:05:30.061060 26092 mini_ranger_kms.cc:240] Using host: 127.25.123.20
I20260928 15:05:30.066128 26092 mini_ranger_kms.cc:276] Ranger KMS PID: 5894
I20260928 15:05:30.066242 26092 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.123.20:35185
Sep 28, 2026 3:05:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:05:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:05:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:05:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:05:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:05:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 15:05:32.440326 26092 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.515s user 0.001s sys 0.183s
I20260928 15:05:32.440482 26092 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:05:32.440531 26092 mini_ranger_kms.cc:401] 127.25.123.20:35185/kms/v1/keys
Sep 28 15:05:32 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607932, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.123.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 15:05:33 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607932, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.123.20@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
Sep 28 15:05:33 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607932, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.123.20@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
Sep 28 15:05:36 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:05:36.217744 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38581
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:38581
--ranger_config_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:36.306252 5945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:36.306486 5945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:36.306510 5945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:36.308104 5945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:36.308161 5945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:36.308175 5945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:36.308188 5945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:36.308200 5945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:36.309986 5945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.123.20:35185/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:38581
--ranger_config_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:36.310278 5945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:36.310559 5945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:36.316479 5951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:36.316529 5945 server_base.cc:1034] running on GCE node
W20260928 15:05:36.316545 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:36.316476 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:36.317318 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:36.317725 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:36.318910 5945 hybrid_clock.cc:648] HybridClock initialized: now 1790607936318830 us; error 102 us; skew 500 ppm
Sep 28 15:05:36 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607936, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:36.328325 5945 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:05:36.328965 5945 webserver.cc:460] Webserver started at http://127.25.123.62:45657/ using document root <none> and password file <none>
I20260928 15:05:36.329236 5945 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:36.329299 5945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:36.329463 5945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:36 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607936, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:36.394377 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "e5a74882b7e4411780c4f3d368587222"
format_stamp: "Formatted at 2026-09-28 15:05:36 on dist-test-slave-qnz0"
server_key: "9ea07ef91ed2226ea41e45675da78d86"
server_key_iv: "d39dc6cf54e083c5d54af4e8a4c23164"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:36.394846 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "e5a74882b7e4411780c4f3d368587222"
format_stamp: "Formatted at 2026-09-28 15:05:36 on dist-test-slave-qnz0"
server_key: "9ea07ef91ed2226ea41e45675da78d86"
server_key_iv: "d39dc6cf54e083c5d54af4e8a4c23164"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:36.396459 5945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 15:05:36.412845 5960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:36.413107 5945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:36.413215 5945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "e5a74882b7e4411780c4f3d368587222"
format_stamp: "Formatted at 2026-09-28 15:05:36 on dist-test-slave-qnz0"
server_key: "9ea07ef91ed2226ea41e45675da78d86"
server_key_iv: "d39dc6cf54e083c5d54af4e8a4c23164"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:36.413342 5945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:36.425589 5945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:36.427696 5945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:36.427857 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:36.433406 5945 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:38581
I20260928 15:05:36.433482 6014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:38581 every 8 connection(s)
I20260928 15:05:36.433902 5945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:36.434589 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 5945
I20260928 15:05:36.434700 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:05:36.434684 6015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:36.434898 26092 external_mini_cluster.cc:1428] Setting key b48a54d334f808448e346f4d778da7ac
I20260928 15:05:36.438233 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Bootstrap starting.
I20260928 15:05:36.439000 6015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:36.439461 6015 log.cc:824] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:36.440832 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: No bootstrap required, opened a new log
Sep 28 15:05:36 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:36.444010 6015 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:36.444232 6015 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:36.444320 6015 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a74882b7e4411780c4f3d368587222, State: Initialized, Role: FOLLOWER
I20260928 15:05:36.444522 6015 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:36.444661 6015 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:36.444708 6015 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:36.444785 6015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:36.445680 6015 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:36.445806 6015 leader_election.cc:302] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a74882b7e4411780c4f3d368587222; no voters:
I20260928 15:05:36.446033 6015 leader_election.cc:288] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:05:36.446099 6020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:36.446343 6020 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 LEADER]: Becoming Leader. State: Replica: e5a74882b7e4411780c4f3d368587222, State: Running, Role: LEADER
I20260928 15:05:36.446465 6020 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:36.446516 6015 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:36.446816 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5a74882b7e4411780c4f3d368587222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } } }
I20260928 15:05:36.446918 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:36.447052 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a74882b7e4411780c4f3d368587222. Latest consensus state: current_term: 1 leader_uuid: "e5a74882b7e4411780c4f3d368587222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } } }
I20260928 15:05:36.447134 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:36.448436 6015 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:05:36.448788 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:36.435467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.25.123.62:38581)
0928 15:05:36.435652 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:36.435660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:36.435691 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:05:36.436381 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:36.436421 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:36.436486 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:36.436637 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:36.437429 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:36.438875 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:36.440057 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:36.440602 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:36.440728 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:36.444199 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:36.444204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:36.444204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:36.444219 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:36.446369 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:36.446840 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:36.446844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:36.446844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:36.446939 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:36.447141 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:36.447143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:36.447143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:36.447532 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:36.447695 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:36.448279 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 15:05:36.448453 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260928 15:05:36.452737 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:36.453531 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:36.458947 6024 catalog_manager.cc:1347] Generated new cluster ID: 095d8179f5924e1c97fdd9db879aa841
I20260928 15:05:36.459038 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:36.471630 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:36.472626 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:36.475538 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Generated new TSK 0
I20260928 15:05:36.475816 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:05:37.412794 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38581
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:37.536646 6061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:37.537047 6061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:37.537133 6061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:37.539163 6061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:37.539314 6061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:37.539404 6061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:05:37.541465 6061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.123.20:35185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:37.541798 6061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:37.542093 6061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:37.547794 6066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.547832 6067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.548702 6070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:37.548800 6061 server_base.cc:1034] running on GCE node
I20260928 15:05:37.549103 6061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:37.549410 6061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:37.550621 6061 hybrid_clock.cc:648] HybridClock initialized: now 1790607937550596 us; error 36 us; skew 500 ppm
Sep 28 15:05:37 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:37.554411 6061 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:05:37.554997 6061 webserver.cc:460] Webserver started at http://127.25.123.1:40643/ using document root <none> and password file <none>
I20260928 15:05:37.555368 6061 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:37.555516 6061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:37.555712 6061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:37 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:37.575289 6061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "cc3816997a594c6a84f3df35b7baa476"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "40034993b5f6f5fe5bd65c29c2cb76c6"
server_key_iv: "3f88a2a3f184eb00a8e3203f1ec4fc34"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:37.575933 6061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "cc3816997a594c6a84f3df35b7baa476"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "40034993b5f6f5fe5bd65c29c2cb76c6"
server_key_iv: "3f88a2a3f184eb00a8e3203f1ec4fc34"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:37.579559 6061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:05:37.596666 6077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.596967 6061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 15:05:37.597160 6061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "cc3816997a594c6a84f3df35b7baa476"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "40034993b5f6f5fe5bd65c29c2cb76c6"
server_key_iv: "3f88a2a3f184eb00a8e3203f1ec4fc34"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:37.597364 6061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:37.609978 6061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:37.612377 6061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:37.612658 6061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:37.613011 6061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:37.613452 6061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:37.613538 6061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.613646 6061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:37.613726 6061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.622367 6061 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:45231
I20260928 15:05:37.622463 6193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:45231 every 8 connection(s)
I20260928 15:05:37.623163 6061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:37.628717 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 6061
I20260928 15:05:37.628829 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:05:37.629007 26092 external_mini_cluster.cc:1428] Setting key 6a2963b99fdcdfd471fc7603e8e15cec
Sep 28 15:05:37 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:37.639699 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:37.626849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38581 (local address 127.25.123.1:32815)
0928 15:05:37.627423 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:37.627474 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:37.628187 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:37.628738 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:37.628743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:37.629035 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:37.629355 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:37.629361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.630388 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.630389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:37.631724 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:37.631728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.632326 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.632327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:37.632505 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:37.633144 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:37.633156 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:37.635671 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:37.637983 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:37.637994 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:37.637997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:37.638309 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:37.638525 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:37.638525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:37.638526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:37.638663 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:37.638976 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:37.638982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:37.639158 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 15:05:37.639266 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:05:37.639435 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":105,"threads_started":1}
I20260928 15:05:37.639988 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:37.627202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:32815 (local address 127.25.123.62:38581)
0928 15:05:37.628382 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:37.628385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:37.628474 (+ 89us) server_negotiation.cc:407] Connection header received
0928 15:05:37.628505 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:37.628506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:37.628519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:37.628585 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:37.629475 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.630273 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.631835 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.632155 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.632231 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:37.635828 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:37.635831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:37.635831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:37.635843 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:37.637867 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:37.638395 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:37.638396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:37.638396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:37.638459 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:37.638728 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:37.638729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:37.638730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:37.638890 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:37.638950 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:37.639766 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 15:05:37.639873 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":65,"threads_started":1}
I20260928 15:05:37.640719 6194 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38581
I20260928 15:05:37.640825 6194 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:37.641023 6194 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:37.641587 5978 ts_manager.cc:194] Registered new tserver with Master: cc3816997a594c6a84f3df35b7baa476 (127.25.123.1:45231)
I20260928 15:05:37.642642 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:32815
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:05:37.672294 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38581
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:37.799477 6200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:37.799813 6200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:37.799852 6200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:37.801853 6200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:37.801939 6200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:37.801982 6200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:05:37.805254 6200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.123.20:35185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:37.805624 6200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:37.805967 6200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:37.813406 6209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.813407 6208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:37.813849 6211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:37.814154 6200 server_base.cc:1034] running on GCE node
I20260928 15:05:37.814399 6200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:37.814718 6200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:37.815903 6200 hybrid_clock.cc:648] HybridClock initialized: now 1790607937815880 us; error 36 us; skew 500 ppm
Sep 28 15:05:37 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:37.819677 6200 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:05:37.820765 6200 webserver.cc:460] Webserver started at http://127.25.123.2:35383/ using document root <none> and password file <none>
I20260928 15:05:37.821095 6200 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:37.821183 6200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:37.821343 6200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:37 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
Sep 28 15:05:37 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:37.836776 6200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "bbe8d7386ec141e382ecabffb9de27b6"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "975984f1274f61049a4de263eda190a1"
server_key_iv: "c883a8dcd87a152bf6cdbd8b871b2ed4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:37.837929 6200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "bbe8d7386ec141e382ecabffb9de27b6"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "975984f1274f61049a4de263eda190a1"
server_key_iv: "c883a8dcd87a152bf6cdbd8b871b2ed4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:37.852928 6200 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.002s sys 0.001s
I20260928 15:05:37.869103 6218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.869407 6200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:05:37.869555 6200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "bbe8d7386ec141e382ecabffb9de27b6"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "975984f1274f61049a4de263eda190a1"
server_key_iv: "c883a8dcd87a152bf6cdbd8b871b2ed4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:37.869728 6200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:37.908972 6200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:37.911641 6200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:37.911895 6200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:37.912192 6200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:37.912674 6200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:37.912767 6200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.912840 6200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:37.912899 6200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:37.919754 6200 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:38325
I20260928 15:05:37.920929 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 6200
I20260928 15:05:37.921047 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:05:37.921267 26092 external_mini_cluster.cc:1428] Setting key bd73aedb0d654b2eb067c849c78bba8b
I20260928 15:05:37.923179 6332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:38325 every 8 connection(s)
I20260928 15:05:37.923334 6200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
Sep 28 15:05:37 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
I20260928 15:05:37.941301 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:37.924494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:54341 (local address 127.25.123.62:38581)
0928 15:05:37.924543 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:37.924546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:37.928783 (+ 4237us) server_negotiation.cc:407] Connection header received
0928 15:05:37.929634 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:37.929636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:37.929656 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:37.929693 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:37.930769 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.931490 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.933126 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.933441 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.933493 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:37.936383 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:37.936385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:37.936386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:37.936416 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:37.938198 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:37.938752 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:37.938754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:37.938754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:37.938820 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:37.939235 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:37.939236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:37.939237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:37.939396 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:37.939443 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:37.941065 (+ 1622us) server_negotiation.cc:299] Negotiation successful
0928 15:05:37.941186 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:05:37.941911 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:37.924646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38581 (local address 127.25.123.2:54341)
0928 15:05:37.928569 (+ 3923us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:37.928653 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:37.929462 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:37.929909 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:37.929914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:37.930191 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:37.930644 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:37.930651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.931639 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.931640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:37.933005 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:37.933009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:37.933626 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:37.933627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:37.933778 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:37.934496 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:37.934507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:37.936249 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:37.938296 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:37.938305 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:37.938309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:37.938642 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:37.938909 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:37.938910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:37.938911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:37.939071 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:37.940641 (+ 1570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:37.940651 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:37.940863 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 15:05:37.941447 (+ 584us) client_negotiation.cc:240] Negotiation successful
0928 15:05:37.941650 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3858,"thread_start_us":97,"threads_started":1}
I20260928 15:05:37.943297 6334 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38581
I20260928 15:05:37.943408 6334 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:37.943636 6334 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:37.944018 5978 ts_manager.cc:194] Registered new tserver with Master: bbe8d7386ec141e382ecabffb9de27b6 (127.25.123.2:38325)
I20260928 15:05:37.944571 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:54341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:05:37.960839 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38581
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:38.090158 6339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:38.090550 6339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:38.090641 6339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:38.092646 6339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:38.092938 6339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:38.093053 6339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:05:38.095785 6339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.123.20:35185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:38.096362 6339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:38.096773 6339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:38.104024 6345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.104257 6344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.104046 6348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:38.105638 6339 server_base.cc:1034] running on GCE node
I20260928 15:05:38.105944 6339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:38.106343 6339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:38.107596 6339 hybrid_clock.cc:648] HybridClock initialized: now 1790607938107571 us; error 39 us; skew 500 ppm
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:38.111232 6339 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:05:38.111801 6339 webserver.cc:460] Webserver started at http://127.25.123.3:37187/ using document root <none> and password file <none>
I20260928 15:05:38.112106 6339 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:38.112164 6339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:38.112298 6339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:38.132756 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "f36f19ca6b894e6fbab64da56981e457"
format_stamp: "Formatted at 2026-09-28 15:05:38 on dist-test-slave-qnz0"
server_key: "f7f240b913fe9ee42a937f02ebaec184"
server_key_iv: "6bef6fe50506906d6a57900fffd960ea"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:38.133358 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "f36f19ca6b894e6fbab64da56981e457"
format_stamp: "Formatted at 2026-09-28 15:05:38 on dist-test-slave-qnz0"
server_key: "f7f240b913fe9ee42a937f02ebaec184"
server_key_iv: "6bef6fe50506906d6a57900fffd960ea"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:38.144990 6339 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260928 15:05:38.163348 6355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.164640 6339 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 15:05:38.164785 6339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "f36f19ca6b894e6fbab64da56981e457"
format_stamp: "Formatted at 2026-09-28 15:05:38 on dist-test-slave-qnz0"
server_key: "f7f240b913fe9ee42a937f02ebaec184"
server_key_iv: "6bef6fe50506906d6a57900fffd960ea"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:38.164902 6339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:38.184744 6339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:38.188087 6339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:38.188257 6339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:38.188649 6339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:38.189054 6339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:38.189102 6339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.189133 6339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:38.189201 6339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.197474 6339 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:33497
I20260928 15:05:38.197647 6471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:33497 every 8 connection(s)
I20260928 15:05:38.198154 6339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:38.208627 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 6339
I20260928 15:05:38.208750 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:05:38.208940 26092 external_mini_cluster.cc:1428] Setting key ddd86a9339d4b4ce00b95528c184ebae
I20260928 15:05:38.213711 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.199176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:49885 (local address 127.25.123.62:38581)
0928 15:05:38.199237 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.199240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.199517 (+ 277us) server_negotiation.cc:407] Connection header received
0928 15:05:38.200494 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.200496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.200513 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.200551 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.201653 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.202473 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.204487 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.204797 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.204851 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.208219 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.208222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.208222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.208232 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.211740 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.212426 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.212427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.212428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.212498 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.212822 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.212824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.212824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.212976 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.213021 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.213482 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.213593 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:05:38.214607 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.199154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38581 (local address 127.25.123.3:49885)
0928 15:05:38.199345 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.199406 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.200286 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.200806 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.200812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.201138 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:38.201496 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.201504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.202584 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.202585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.204222 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.204231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.204986 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.204987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.205201 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.205955 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:38.205968 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:38.208044 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:38.211872 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.211886 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.211889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.212292 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.212582 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.212583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.212583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.212737 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.213070 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:38.213091 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:38.213303 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.214142 (+ 839us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.214340 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260928 15:05:38.217453 6472 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38581
I20260928 15:05:38.217582 6472 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.217803 6472 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:38.218319 5978 ts_manager.cc:194] Registered new tserver with Master: f36f19ca6b894e6fbab64da56981e457 (127.25.123.3:33497)
I20260928 15:05:38.218871 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:49885
I20260928 15:05:38.220985 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:38.221726 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 6061
I20260928 15:05:38.231209 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 6200
I20260928 15:05:38.239437 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 6339
I20260928 15:05:38.247355 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 5945
I20260928 15:05:38.255587 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38581
--webserver_interface=127.25.123.62
--webserver_port=45657
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:38581
--ranger_config_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:38.348322 6475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:38.348686 6475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:38.348728 6475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:38.350271 6475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:38.350383 6475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:38.350414 6475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:38.350443 6475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:38.350474 6475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:38.352350 6475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:38581
--ranger_config_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=45657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:38.352691 6475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:38.352946 6475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:38.358343 6480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.358321 6481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.358490 6483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:38.358623 6475 server_base.cc:1034] running on GCE node
I20260928 15:05:38.358932 6475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:38.359330 6475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:38.360641 6475 hybrid_clock.cc:648] HybridClock initialized: now 1790607938360609 us; error 49 us; skew 500 ppm
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:38.363518 6475 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:05:38.364099 6475 webserver.cc:460] Webserver started at http://127.25.123.62:45657/ using document root <none> and password file <none>
I20260928 15:05:38.364342 6475 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:38.364444 6475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:38.435243 6475 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:38.436136 6491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.436300 6475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:38.436436 6475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "e5a74882b7e4411780c4f3d368587222"
format_stamp: "Formatted at 2026-09-28 15:05:36 on dist-test-slave-qnz0"
server_key: "9ea07ef91ed2226ea41e45675da78d86"
server_key_iv: "d39dc6cf54e083c5d54af4e8a4c23164"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:38.436836 6475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:38.480978 6475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:38.483318 6475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:38.483762 6475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:38.488744 6475 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:38581
I20260928 15:05:38.488785 6543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:38581 every 8 connection(s)
I20260928 15:05:38.489192 6475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:38.490473 6544 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:05:38.491106 6544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Bootstrap starting.
I20260928 15:05:38.494737 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 6475
I20260928 15:05:38.495074 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:45231
--local_ip_for_outbound_sockets=127.25.123.1
--tserver_master_addrs=127.25.123.62:38581
--webserver_port=40643
--webserver_interface=127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:38.500505 6544 log.cc:824] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:38.503118 6544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:38.503492 6544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Bootstrap complete.
I20260928 15:05:38.507747 6544 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:38.508116 6544 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a74882b7e4411780c4f3d368587222, State: Initialized, Role: FOLLOWER
I20260928 15:05:38.508340 6544 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:38.508435 6544 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:38.508476 6544 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:38.508525 6544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:05:38.509931 6544 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:38.510052 6544 leader_election.cc:302] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a74882b7e4411780c4f3d368587222; no voters:
I20260928 15:05:38.510226 6544 leader_election.cc:288] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:05:38.510358 6548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:05:38.510490 6544 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:38.510571 6548 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [term 2 LEADER]: Becoming Leader. State: Replica: e5a74882b7e4411780c4f3d368587222, State: Running, Role: LEADER
I20260928 15:05:38.510726 6548 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } }
I20260928 15:05:38.511096 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5a74882b7e4411780c4f3d368587222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } } }
I20260928 15:05:38.511189 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:38.511334 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a74882b7e4411780c4f3d368587222. Latest consensus state: current_term: 2 leader_uuid: "e5a74882b7e4411780c4f3d368587222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a74882b7e4411780c4f3d368587222" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38581 } } }
I20260928 15:05:38.511392 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:38.511390 6557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:38.512045 6557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:38.512235 6557 catalog_manager.cc:1259] Loaded cluster ID: 095d8179f5924e1c97fdd9db879aa841
I20260928 15:05:38.512307 6557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:38.512451 6544 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:05:38.517306 6557 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:38.518070 6557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Loaded TSK: 0
I20260928 15:05:38.518481 6557 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:05:38.623514 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:38.623831 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:38.623870 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:38.625838 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:38.625931 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:38.625980 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:05:38.628822 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:45231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=40643
--tserver_master_addrs=127.25.123.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:38.629410 6546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:38.629736 6546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:38.636322 6585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.636375 6582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.636842 6583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:38.637025 6546 server_base.cc:1034] running on GCE node
I20260928 15:05:38.637341 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:38.637668 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:38.638856 6546 hybrid_clock.cc:648] HybridClock initialized: now 1790607938638824 us; error 46 us; skew 500 ppm
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:38.642506 6546 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:05:38.643126 6546 webserver.cc:460] Webserver started at http://127.25.123.1:40643/ using document root <none> and password file <none>
I20260928 15:05:38.643431 6546 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:38.643519 6546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:38.676374 6546 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 15:05:38.677522 6594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.677811 6546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:05:38.677903 6546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "cc3816997a594c6a84f3df35b7baa476"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "40034993b5f6f5fe5bd65c29c2cb76c6"
server_key_iv: "3f88a2a3f184eb00a8e3203f1ec4fc34"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:38.678351 6546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:38.691296 6546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:38.694486 6546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:38.694666 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:38.694947 6546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:38.695358 6546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:38.695463 6546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.695574 6546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:38.695662 6546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.703609 6546 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:45231
I20260928 15:05:38.703693 6707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:45231 every 8 connection(s)
I20260928 15:05:38.704166 6546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:38.705582 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 6546
I20260928 15:05:38.705896 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:38325
--local_ip_for_outbound_sockets=127.25.123.2
--tserver_master_addrs=127.25.123.62:38581
--webserver_port=35383
--webserver_interface=127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:38.717890 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.704960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38581 (local address 127.25.123.1:50527)
0928 15:05:38.705469 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.705521 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.706380 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.707044 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.707049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.707316 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:38.707539 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.707544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.709324 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.709325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.710223 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.710227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.711075 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.711077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.711218 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.711870 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:38.711880 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:38.713636 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:38.715841 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.715852 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.715855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.716148 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.716509 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.716510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.716511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.716644 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.717085 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:38.717091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:38.717291 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.717431 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.717633 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":112,"threads_started":1}
I20260928 15:05:38.719045 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.705131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:50527 (local address 127.25.123.62:38581)
0928 15:05:38.705343 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.705351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.705655 (+ 304us) server_negotiation.cc:407] Connection header received
0928 15:05:38.706733 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.706742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.706809 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.706975 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.707673 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.709178 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.710352 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.710955 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.711083 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.713789 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.713795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.713795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.713809 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.715714 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.716262 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.716267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.716267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.716372 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.716716 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.716718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.716718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.716995 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.717096 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.718658 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.718834 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260928 15:05:38.720633 6708 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38581
I20260928 15:05:38.720753 6708 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.721004 6708 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
W20260928 15:05:38.722047 6708 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:38581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:05:38.722128 6708 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.722224 6708 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:38.722720 6708 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.722797 6708 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
W20260928 15:05:38.723181 6708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:05:38.836567 6712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:38.836971 6712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:38.837080 6712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:38.839114 6712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:38.839253 6712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:38.839361 6712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:05:38.842213 6712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:38325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=35383
--tserver_master_addrs=127.25.123.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:38.842516 6712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:38.842923 6712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:38.849427 6720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.849308 6718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:38.849313 6717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:38.849956 6712 server_base.cc:1034] running on GCE node
I20260928 15:05:38.850201 6712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:38.850562 6712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:38.851838 6712 hybrid_clock.cc:648] HybridClock initialized: now 1790607938851806 us; error 47 us; skew 500 ppm
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:38.855592 6712 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:05:38.856308 6712 webserver.cc:460] Webserver started at http://127.25.123.2:35383/ using document root <none> and password file <none>
I20260928 15:05:38.856657 6712 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:38.856820 6712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:38.880128 6712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 15:05:38.881253 6729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.881484 6712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:05:38.881593 6712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "bbe8d7386ec141e382ecabffb9de27b6"
format_stamp: "Formatted at 2026-09-28 15:05:37 on dist-test-slave-qnz0"
server_key: "975984f1274f61049a4de263eda190a1"
server_key_iv: "c883a8dcd87a152bf6cdbd8b871b2ed4"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:38.882036 6712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:38.919080 6712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:38.921643 6712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:38.921831 6712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:38.922215 6712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:38.922699 6712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:38.922797 6712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.922888 6712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:38.922941 6712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:38.930824 6712 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:38325
I20260928 15:05:38.931510 6712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:38.932009 6843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:38325 every 8 connection(s)
I20260928 15:05:38.937256 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 6712
I20260928 15:05:38.937700 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.123.20:35185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:33497
--local_ip_for_outbound_sockets=127.25.123.3
--tserver_master_addrs=127.25.123.62:38581
--webserver_port=37187
--webserver_interface=127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:05:38 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:38.948640 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.932514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:37639 (local address 127.25.123.62:38581)
0928 15:05:38.932572 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:38.932575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:38.936861 (+ 4286us) server_negotiation.cc:407] Connection header received
0928 15:05:38.937639 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:38.937643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:38.937672 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:38.937858 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:38.939132 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.939888 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.941538 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.941784 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.941928 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.944827 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:38.944830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:38.944831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:38.944841 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:38.946847 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.947402 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.947403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.947404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.947470 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:38.947829 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:38.947830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:38.947831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:38.947971 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:38.948056 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:38.948422 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 15:05:38.948530 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 15:05:38.948963 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:38.932519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38581 (local address 127.25.123.2:37639)
0928 15:05:38.936578 (+ 4059us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:38.936665 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:38.937444 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:38.938285 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:38.938292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:38.938586 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:38.938930 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.938937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.940015 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.940017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.941394 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:38.941397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:38.941972 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:38.941974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:38.942126 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:38.942872 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:38.942884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:38.944688 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:38.946966 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.946977 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.946980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.947298 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.947550 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:38.947551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:38.947552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:38.947741 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:38.948045 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:38.948053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:38.948238 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 15:05:38.948359 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 15:05:38.948589 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3985,"thread_start_us":91,"threads_started":1}
I20260928 15:05:38.949542 6844 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38581
I20260928 15:05:38.949685 6844 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.949914 6844 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
W20260928 15:05:38.950675 6844 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:38581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:05:38.950786 6844 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.950872 6844 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:38.951342 6844 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:38.951440 6844 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
W20260928 15:05:38.951784 6844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:05:39.074177 6847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:39.074507 6847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:39.074554 6847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:39.076519 6847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:39.076668 6847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:39.076766 6847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:05:39.079347 6847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.123.20:35185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:33497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=37187
--tserver_master_addrs=127.25.123.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:39.079929 6847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:39.080343 6847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:39.088001 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:39.087965 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:39.088467 6856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:39.088773 6847 server_base.cc:1034] running on GCE node
I20260928 15:05:39.089010 6847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:39.089315 6847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:39.092578 6847 hybrid_clock.cc:648] HybridClock initialized: now 1790607939092535 us; error 57 us; skew 500 ppm
Sep 28 15:05:39 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:39.096277 6847 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:05:39.096948 6847 webserver.cc:460] Webserver started at http://127.25.123.3:37187/ using document root <none> and password file <none>
I20260928 15:05:39.097234 6847 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:39.097326 6847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:05:39 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:39.121070 6847 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:05:39.122350 6864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:39.123207 6847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:05:39.123327 6847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "f36f19ca6b894e6fbab64da56981e457"
format_stamp: "Formatted at 2026-09-28 15:05:38 on dist-test-slave-qnz0"
server_key: "f7f240b913fe9ee42a937f02ebaec184"
server_key_iv: "6bef6fe50506906d6a57900fffd960ea"
server_key_version: "kuduclusterkey@0"
I20260928 15:05:39.123872 6847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:39.135257 6847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:39.138289 6847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:39.138451 6847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:39.138724 6847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:39.139117 6847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:39.139189 6847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:39.139233 6847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:39.139251 6847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:39.147276 6847 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:33497
I20260928 15:05:39.147478 6977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:33497 every 8 connection(s)
I20260928 15:05:39.147925 6847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:39.153369 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 6847
Sep 28 15:05:39 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:39.161612 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:39.148823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:38581 (local address 127.25.123.3:48389)
0928 15:05:39.149025 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:39.149097 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:39.149780 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:39.150157 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:39.150161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:39.150409 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:39.150731 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:39.150738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.151875 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.151876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:39.153308 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:39.153312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.153937 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.153938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:39.154110 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:39.154968 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:39.154987 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:39.157467 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:39.159594 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:39.159607 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:39.159611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:39.160007 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:39.160263 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:39.160264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:39.160265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:39.160461 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:39.160815 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:39.160822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:39.161023 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 15:05:39.161143 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 15:05:39.161334 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260928 15:05:39.162165 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:39.148860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:48389 (local address 127.25.123.62:38581)
0928 15:05:39.148946 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:39.148950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:39.149216 (+ 266us) server_negotiation.cc:407] Connection header received
0928 15:05:39.149959 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:39.149961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:39.149979 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:39.150066 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:39.150848 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.151703 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.153456 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.153750 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.153807 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:39.157591 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:39.157593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:39.157593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:39.157603 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:39.159485 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:39.160108 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:39.160110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:39.160110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:39.160180 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:39.160546 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:39.160548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:39.160548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:39.160698 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:39.160783 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:39.161972 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0928 15:05:39.162072 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:05:39.162743 6978 heartbeater.cc:347] Connected to a master server at 127.25.123.62:38581
I20260928 15:05:39.162921 6978 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:39.163193 6978 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
W20260928 15:05:39.163949 6978 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:38581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:05:39.164083 6978 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:39.164163 6978 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:39.165038 6978 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:39.165184 6978 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
W20260928 15:05:39.165562 6978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:05:39.170758 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:39.153879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.25.123.62:38581)
0928 15:05:39.156042 (+ 2163us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:39.156044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:39.156052 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:05:39.156079 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:39.156080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:39.156094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:39.156180 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:39.157100 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.163059 (+ 5959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.164009 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:39.164466 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:39.164653 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:39.166480 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:39.166482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:39.166482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:39.166488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:39.169199 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:39.169671 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:39.169673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:39.169673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:39.169736 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:39.170149 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:39.170150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:39.170150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:39.170305 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:39.170488 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:39.170536 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 15:05:39.170639 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2135,"thread_start_us":67,"threads_started":1}
I20260928 15:05:39.723457 6708 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:39.723735 6708 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:39.724503 6508 ts_manager.cc:194] Registered new tserver with Master: cc3816997a594c6a84f3df35b7baa476 (127.25.123.1:45231)
I20260928 15:05:39.725792 6508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:50527
Sep 28 15:05:39 dist-test-slave-qnz0 krb5kdc[5497](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:39.952013 6844 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:39.952188 6844 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:39.952505 6508 ts_manager.cc:194] Registered new tserver with Master: bbe8d7386ec141e382ecabffb9de27b6 (127.25.123.2:38325)
I20260928 15:05:39.953073 6508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:37639
I20260928 15:05:40.165824 6978 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:40.165988 6978 heartbeater.cc:511] Master 127.25.123.62:38581 requested a full tablet report, sending...
I20260928 15:05:40.166416 6508 ts_manager.cc:194] Registered new tserver with Master: f36f19ca6b894e6fbab64da56981e457 (127.25.123.3:33497)
I20260928 15:05:40.166935 6508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:48389
I20260928 15:05:40.173775 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:40.186749 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.174860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.25.123.62:38581)
0928 15:05:40.175388 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:40.175391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:40.177450 (+ 2059us) server_negotiation.cc:407] Connection header received
0928 15:05:40.177751 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:40.177753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:40.177768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:40.177831 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:40.178567 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.179392 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.180730 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.181059 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.181123 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.182689 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:40.182691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:40.182692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:40.182701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:40.184595 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:40.185234 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:40.185236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:40.185237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:40.185306 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:40.185512 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:40.185512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:40.185513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:40.185684 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:40.186017 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:40.186543 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 15:05:40.186637 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":72,"threads_started":1}
Sep 28 15:05:40 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:40.188961 6508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:05:40.191241 6508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:05:40.206753 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.195056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:33497 (local address 127.0.0.1:39722)
0928 15:05:40.200664 (+ 5608us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:40.200739 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:40.200834 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:40.201192 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:40.201196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:40.201198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:40.201302 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.201304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.202507 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.202509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.204019 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.204023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.206057 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.206059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.206303 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.206336 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:05:40.206548 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 15:05:40.206647 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5555,"thread_start_us":74,"threads_started":1}
I20260928 15:05:40.207196 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.195492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.25.123.3:33497)
0928 15:05:40.200537 (+ 5045us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:40.200546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:40.200930 (+ 384us) server_negotiation.cc:407] Connection header received
0928 15:05:40.200989 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:40.200992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:40.201011 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:40.201085 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:40.201397 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.202393 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.204131 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.205854 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.205937 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.206015 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:40.206997 (+ 982us) server_negotiation.cc:299] Negotiation successful
0928 15:05:40.207061 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4990,"thread_start_us":73,"threads_started":1}
I20260928 15:05:40.207998 6912 tablet_service.cc:1467] Processing CreateTablet for tablet 74fb046de8804b649d0820a99517ccfd (DEFAULT_TABLE table=test-table [id=fdcde5bd17fb4972997c27c4e4073742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:40.208442 6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fb046de8804b649d0820a99517ccfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:40.219673 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.195432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:38325 (local address 127.0.0.1:52434)
0928 15:05:40.202704 (+ 7272us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:40.202712 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:40.202746 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:40.214423 (+ 11677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:40.214425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:40.214430 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:40.214514 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.214516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.215894 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.215895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.217130 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.217133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.219358 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.219360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.219437 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.219444 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:05:40.219510 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:05:40.219566 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7260,"thread_start_us":51,"threads_started":1}
I20260928 15:05:40.220047 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.195846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.25.123.2:38325)
0928 15:05:40.212538 (+ 16692us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:40.212544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:40.212560 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:05:40.212600 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:40.212603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:40.212621 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:40.212716 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:40.214629 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.215776 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.217234 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.218080 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.218156 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.218222 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:40.219912 (+ 1690us) server_negotiation.cc:299] Negotiation successful
0928 15:05:40.219965 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16634,"thread_start_us":68,"threads_started":1}
I20260928 15:05:40.220875 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.195290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:45231 (local address 127.0.0.1:41106)
0928 15:05:40.202558 (+ 7268us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:40.202571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:40.202625 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:40.203249 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:40.203250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:40.203254 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:40.203317 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.203318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.210666 (+ 7348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.210668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.219382 (+ 8714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.219386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.220608 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.220610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.220700 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.220704 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:05:40.220751 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 15:05:40.220795 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7244,"thread_start_us":68,"threads_started":1}
I20260928 15:05:40.221122 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.195695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.25.123.1:45231)
0928 15:05:40.202870 (+ 7175us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:40.202876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:40.202888 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:05:40.202927 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:40.202929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:40.202946 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:40.203024 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:40.205241 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.209156 (+ 3915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.219520 (+ 10364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.220376 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.220466 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.220543 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:40.220988 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 15:05:40.221041 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7123,"thread_start_us":55,"threads_started":1}
I20260928 15:05:40.221027 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 74fb046de8804b649d0820a99517ccfd (DEFAULT_TABLE table=test-table [id=fdcde5bd17fb4972997c27c4e4073742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:40.221460 7011 tablet_bootstrap.cc:492] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Bootstrap starting.
I20260928 15:05:40.222283 7011 tablet_bootstrap.cc:654] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:40.222589 7011 log.cc:824] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:40.221458 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fb046de8804b649d0820a99517ccfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:40.224049 6642 tablet_service.cc:1467] Processing CreateTablet for tablet 74fb046de8804b649d0820a99517ccfd (DEFAULT_TABLE table=test-table [id=fdcde5bd17fb4972997c27c4e4073742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:40.224457 6642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fb046de8804b649d0820a99517ccfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:40.228345 7011 tablet_bootstrap.cc:492] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: No bootstrap required, opened a new log
I20260928 15:05:40.228511 7011 ts_tablet_manager.cc:1397] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 15:05:40.231218 7011 raft_consensus.cc:348] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.231472 7011 raft_consensus.cc:374] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:40.231643 7011 raft_consensus.cc:729] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f19ca6b894e6fbab64da56981e457, State: Initialized, Role: FOLLOWER
I20260928 15:05:40.231791 7011 consensus_queue.cc:260] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.232106 7011 ts_tablet_manager.cc:1428] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 15:05:40.232429 7013 tablet_bootstrap.cc:492] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Bootstrap starting.
I20260928 15:05:40.232481 6978 heartbeater.cc:503] Master 127.25.123.62:38581 was elected leader, sending a full tablet report...
I20260928 15:05:40.233202 7013 tablet_bootstrap.cc:654] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:40.233608 7013 log.cc:824] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:40.234041 7014 tablet_bootstrap.cc:492] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: Bootstrap starting.
I20260928 15:05:40.234758 7014 tablet_bootstrap.cc:654] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:40.234997 7013 tablet_bootstrap.cc:492] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: No bootstrap required, opened a new log
I20260928 15:05:40.235117 7013 ts_tablet_manager.cc:1397] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:05:40.235111 7014 log.cc:824] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:40.240528 7013 raft_consensus.cc:348] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.240906 7013 raft_consensus.cc:374] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:40.241037 7013 raft_consensus.cc:729] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe8d7386ec141e382ecabffb9de27b6, State: Initialized, Role: FOLLOWER
I20260928 15:05:40.241303 7013 consensus_queue.cc:260] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.241758 7013 ts_tablet_manager.cc:1428] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260928 15:05:40.241953 6844 heartbeater.cc:503] Master 127.25.123.62:38581 was elected leader, sending a full tablet report...
I20260928 15:05:40.250111 7014 tablet_bootstrap.cc:492] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: No bootstrap required, opened a new log
I20260928 15:05:40.250329 7014 ts_tablet_manager.cc:1397] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260928 15:05:40.255239 7014 raft_consensus.cc:348] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.255594 7014 raft_consensus.cc:374] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:40.255697 7014 raft_consensus.cc:729] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3816997a594c6a84f3df35b7baa476, State: Initialized, Role: FOLLOWER
I20260928 15:05:40.255952 7014 consensus_queue.cc:260] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.256373 7014 ts_tablet_manager.cc:1428] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260928 15:05:40.256608 6708 heartbeater.cc:503] Master 127.25.123.62:38581 was elected leader, sending a full tablet report...
W20260928 15:05:40.399281 6979 tablet.cc:2369] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:40.410235 7015 raft_consensus.cc:482] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:40.410408 7015 raft_consensus.cc:504] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.410830 7015 leader_election.cc:288] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc3816997a594c6a84f3df35b7baa476 (127.25.123.1:45231), bbe8d7386ec141e382ecabffb9de27b6 (127.25.123.2:38325)
I20260928 15:05:40.415297 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.411139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:47095 (local address 127.25.123.1:45231)
0928 15:05:40.411184 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:40.411187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:40.411399 (+ 212us) server_negotiation.cc:407] Connection header received
0928 15:05:40.411661 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:40.411662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:40.411674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:40.411704 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:40.411977 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.412729 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.414344 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.414893 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.414944 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.414955 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:40.415173 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 15:05:40.415217 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:05:40.415402 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.411542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:39007 (local address 127.25.123.2:38325)
0928 15:05:40.411585 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:40.411587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:40.411595 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:05:40.411687 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:40.411688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:40.411699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:40.411730 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:40.411973 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.412729 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.414350 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.414935 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.414991 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.415002 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:40.415273 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 15:05:40.415324 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:05:40.415637 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.411311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:38325 (local address 127.25.123.3:39007)
0928 15:05:40.411529 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:40.411540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:40.411594 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:40.411807 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:40.411807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:40.411810 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:40.411883 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.411885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.412842 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.412843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.414231 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.414234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.415069 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.415070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.415185 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.415189 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:05:40.415490 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 15:05:40.415538 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":83,"thread_start_us":88,"threads_started":1}
I20260928 15:05:40.416033 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.411059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:45231 (local address 127.25.123.3:47095)
0928 15:05:40.411314 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:40.411326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:40.411428 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:40.411791 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:40.411791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:40.411793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:40.411903 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.411905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.412850 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.412851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.414233 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:40.414236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.415027 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.415028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:40.415096 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.415099 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:05:40.415814 (+ 715us) client_negotiation.cc:240] Negotiation successful
0928 15:05:40.415941 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":53,"thread_start_us":129,"threads_started":1}
I20260928 15:05:40.416093 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fb046de8804b649d0820a99517ccfd" candidate_uuid: "f36f19ca6b894e6fbab64da56981e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe8d7386ec141e382ecabffb9de27b6" is_pre_election: true
I20260928 15:05:40.416298 6798 raft_consensus.cc:2454] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f19ca6b894e6fbab64da56981e457 in term 0.
I20260928 15:05:40.416549 6662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fb046de8804b649d0820a99517ccfd" candidate_uuid: "f36f19ca6b894e6fbab64da56981e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3816997a594c6a84f3df35b7baa476" is_pre_election: true
I20260928 15:05:40.416627 6866 leader_election.cc:302] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbe8d7386ec141e382ecabffb9de27b6, f36f19ca6b894e6fbab64da56981e457; no voters:
I20260928 15:05:40.416689 6662 raft_consensus.cc:2454] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36f19ca6b894e6fbab64da56981e457 in term 0.
I20260928 15:05:40.416775 7015 raft_consensus.cc:2781] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:40.416826 7015 raft_consensus.cc:482] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:40.416850 7015 raft_consensus.cc:3037] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:40.417923 7015 raft_consensus.cc:504] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.418088 7015 leader_election.cc:288] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [CANDIDATE]: Term 1 election: Requested vote from peers cc3816997a594c6a84f3df35b7baa476 (127.25.123.1:45231), bbe8d7386ec141e382ecabffb9de27b6 (127.25.123.2:38325)
I20260928 15:05:40.418355 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fb046de8804b649d0820a99517ccfd" candidate_uuid: "f36f19ca6b894e6fbab64da56981e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe8d7386ec141e382ecabffb9de27b6"
I20260928 15:05:40.418356 6662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fb046de8804b649d0820a99517ccfd" candidate_uuid: "f36f19ca6b894e6fbab64da56981e457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3816997a594c6a84f3df35b7baa476"
I20260928 15:05:40.418445 6662 raft_consensus.cc:3037] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:40.418445 6798 raft_consensus.cc:3037] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:40.419514 6662 raft_consensus.cc:2454] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f19ca6b894e6fbab64da56981e457 in term 1.
I20260928 15:05:40.419468 6798 raft_consensus.cc:2454] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36f19ca6b894e6fbab64da56981e457 in term 1.
I20260928 15:05:40.419723 6866 leader_election.cc:302] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc3816997a594c6a84f3df35b7baa476, f36f19ca6b894e6fbab64da56981e457; no voters:
I20260928 15:05:40.419835 7015 raft_consensus.cc:2781] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:40.420051 7015 raft_consensus.cc:686] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 1 LEADER]: Becoming Leader. State: Replica: f36f19ca6b894e6fbab64da56981e457, State: Running, Role: LEADER
I20260928 15:05:40.420279 7015 consensus_queue.cc:237] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } }
I20260928 15:05:40.421861 6508 catalog_manager.cc:5495] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 reported cstate change: term changed from 0 to 1, leader changed from <none> to f36f19ca6b894e6fbab64da56981e457 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "f36f19ca6b894e6fbab64da56981e457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f36f19ca6b894e6fbab64da56981e457" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 } health_report { overall_health: UNKNOWN } } }
W20260928 15:05:40.433418 6845 tablet.cc:2369] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:05:40.455928 6709 tablet.cc:2369] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:05:40 dist-test-slave-qnz0 krb5kdc[5497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607939, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for HTTP/127.25.123.20@KRBTEST.COM
I20260928 15:05:40.483047 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.479458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.25.123.3:33497)
0928 15:05:40.479516 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:40.479519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:40.479526 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:05:40.479568 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:40.479569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:40.479591 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:40.479671 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:40.479816 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.480576 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.481704 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:40.482018 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:40.482103 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:40.482340 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:40.482503 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:40.482583 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:40.482926 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 15:05:40.482953 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:05:40.486023 6798 raft_consensus.cc:1261] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 1 FOLLOWER]: Refusing update from remote peer f36f19ca6b894e6fbab64da56981e457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:40.486505 7015 consensus_queue.cc:1035] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe8d7386ec141e382ecabffb9de27b6" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:40.489164 6662 raft_consensus.cc:1261] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476 [term 1 FOLLOWER]: Refusing update from remote peer f36f19ca6b894e6fbab64da56981e457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:40.489598 7015 consensus_queue.cc:1035] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3816997a594c6a84f3df35b7baa476" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 45231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:40.494019 7026 mvcc.cc:205] Tried to move back new op lower bound from 7334330124226789376 to 7334330123962408960. Current Snapshot: MvccSnapshot[applied={T|T < 7334330124226789376}]
I20260928 15:05:40.497483 7029 mvcc.cc:205] Tried to move back new op lower bound from 7334330124226789376 to 7334330123962408960. Current Snapshot: MvccSnapshot[applied={T|T < 7334330124226789376}]
I20260928 15:05:40.498052 7027 mvcc.cc:205] Tried to move back new op lower bound from 7334330124226789376 to 7334330123962408960. Current Snapshot: MvccSnapshot[applied={T|T < 7334330124226789376}]
I20260928 15:05:40.500499 6508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:40.500651 6508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:40.501592 6508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5a74882b7e4411780c4f3d368587222: Sending DeleteTablet for 3 replicas of tablet 74fb046de8804b649d0820a99517ccfd
I20260928 15:05:40.502179 6912 tablet_service.cc:1514] Processing DeleteTablet for tablet 74fb046de8804b649d0820a99517ccfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:39722
I20260928 15:05:40.502465 6642 tablet_service.cc:1514] Processing DeleteTablet for tablet 74fb046de8804b649d0820a99517ccfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:41106
I20260928 15:05:40.502702 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet 74fb046de8804b649d0820a99517ccfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:40 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:52434
I20260928 15:05:40.503443 7036 tablet_replica.cc:330] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: stopping tablet replica
I20260928 15:05:40.503533 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 6546
I20260928 15:05:40.503695 7038 tablet_replica.cc:330] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: stopping tablet replica
I20260928 15:05:40.503810 7037 tablet_replica.cc:330] T 74fb046de8804b649d0820a99517ccfd P cc3816997a594c6a84f3df35b7baa476: stopping tablet replica
I20260928 15:05:40.503983 7038 raft_consensus.cc:2229] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:40.504321 7036 raft_consensus.cc:2229] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:05:40.504565 7038 raft_consensus.cc:2258] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:40.504777 7036 raft_consensus.cc:2258] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:40.505475 7036 ts_tablet_manager.cc:1905] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:40.505900 7038 ts_tablet_manager.cc:1905] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:40.510051 7038 ts_tablet_manager.cc:1918] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:40.510243 7038 log.cc:1196] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/74fb046de8804b649d0820a99517ccfd
I20260928 15:05:40.510599 7036 ts_tablet_manager.cc:1918] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:40.510756 7036 log.cc:1196] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/74fb046de8804b649d0820a99517ccfd
I20260928 15:05:40.511118 7036 ts_tablet_manager.cc:1939] T 74fb046de8804b649d0820a99517ccfd P f36f19ca6b894e6fbab64da56981e457: Deleting consensus metadata
I20260928 15:05:40.511680 6494 catalog_manager.cc:4841] TS f36f19ca6b894e6fbab64da56981e457 (127.25.123.3:33497): tablet 74fb046de8804b649d0820a99517ccfd (table test-table [id=fdcde5bd17fb4972997c27c4e4073742]) successfully deleted
I20260928 15:05:40.512001 7038 ts_tablet_manager.cc:1939] T 74fb046de8804b649d0820a99517ccfd P bbe8d7386ec141e382ecabffb9de27b6: Deleting consensus metadata
I20260928 15:05:40.512540 6493 catalog_manager.cc:4841] TS bbe8d7386ec141e382ecabffb9de27b6 (127.25.123.2:38325): tablet 74fb046de8804b649d0820a99517ccfd (table test-table [id=fdcde5bd17fb4972997c27c4e4073742]) successfully deleted
W20260928 15:05:40.514431 6493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:45231 (error 108)
I20260928 15:05:40.514721 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 6712
I20260928 15:05:40.520645 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:40.516429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:45231 (local address 127.0.0.1:41112)
0928 15:05:40.520472 (+ 4043us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:40.520589 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4002}
W20260928 15:05:40.521478 6493 catalog_manager.cc:4584] TS cc3816997a594c6a84f3df35b7baa476 (127.25.123.1:45231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74fb046de8804b649d0820a99517ccfd: Network error: Client connection negotiation failed: client connection to 127.25.123.1:45231: connect: Connection refused (error 111)
I20260928 15:05:40.522351 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 6847
I20260928 15:05:40.529109 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 6475
I20260928 15:05:40.538060 26092 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:05:40.883958 26092 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:05:40.884069 26092 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:05:41.232735 26092 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:05:41.232 UTC [5527] LOG: received smart shutdown request
2026-09-28 15:05:41.238 UTC [5527] LOG: background worker "logical replication launcher" (PID 5536) exited with exit code 1
2026-09-28 15:05:41.238 UTC [5531] LOG: shutting down
2026-09-28 15:05:41.491 UTC [5527] LOG: database system is shut down
2026-09-28T15:05:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59479 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:05:41 dist-test-slave-qnz0 krb5kdc[7049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:05:41 dist-test-slave-qnz0 krb5kdc[7049](info): set up 2 sockets
Sep 28 15:05:41 dist-test-slave-qnz0 krb5kdc[7049](info): commencing operation
krb5kdc: starting...
W20260928 15:05:42.720089 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:05:42 dist-test-slave-qnz0 krb5kdc[7049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:05:42.869128 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39681
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39681,127.25.123.61:46633,127.25.123.60:43609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:42.967384 7062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:42.967748 7062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:42.967803 7062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:42.969301 7062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:42.969404 7062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:42.969431 7062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:42.969456 7062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:42.969491 7062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:42.971487 7062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39681,127.25.123.61:46633,127.25.123.60:43609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:39681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:42.971899 7062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:42.972203 7062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:42.979485 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:42.979589 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:42.979632 7062 server_base.cc:1034] running on GCE node
W20260928 15:05:42.979519 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:42.980490 7062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:42.981187 7062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:42.982420 7062 hybrid_clock.cc:648] HybridClock initialized: now 1790607942982373 us; error 67 us; skew 500 ppm
Sep 28 15:05:42 dist-test-slave-qnz0 krb5kdc[7049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:42.986508 7062 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:05:42.988304 7062 webserver.cc:460] Webserver started at http://127.25.123.62:33371/ using document root <none> and password file <none>
I20260928 15:05:42.988833 7062 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:42.988927 7062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:42.989132 7062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:42.990335 7062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "215486b95b3b4b768f75111d000b1878"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-qnz0"
server_key: "a9312a470147f55ee82d8dc1fb302e32"
server_key_iv: "a81b37d3ab5382740005eaf57419d5a2"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.990748 7062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "215486b95b3b4b768f75111d000b1878"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-qnz0"
server_key: "a9312a470147f55ee82d8dc1fb302e32"
server_key_iv: "a81b37d3ab5382740005eaf57419d5a2"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.992323 7062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 15:05:42.993911 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:42.994239 7062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:05:42.994380 7062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "215486b95b3b4b768f75111d000b1878"
format_stamp: "Formatted at 2026-09-28 15:05:42 on dist-test-slave-qnz0"
server_key: "a9312a470147f55ee82d8dc1fb302e32"
server_key_iv: "a81b37d3ab5382740005eaf57419d5a2"
server_key_version: "encryptionkey@0"
I20260928 15:05:42.994483 7062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:43.019353 7062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:43.022338 7062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:43.022593 7062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:43.027524 7062 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:39681
I20260928 15:05:43.027627 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:39681 every 8 connection(s)
I20260928 15:05:43.028014 7062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:43.029202 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:43.030808 7130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.036602 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7062
I20260928 15:05:43.036775 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:05:43.037024 26092 external_mini_cluster.cc:1428] Setting key 831b006d2b6ddf74c207a7ebd11a0418
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:43.042654 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.031593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39681 (local address 127.0.0.1:43302)
0928 15:05:43.031879 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.031931 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.032651 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.032989 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.032992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.033338 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.033516 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.033520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.034742 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.034744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.035524 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.035526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.036129 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.036130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.036204 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.036945 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.036955 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.038669 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.040858 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.040865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.040867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.041095 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.041380 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.041380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.041381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.041481 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.041937 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.041942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.042017 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.042144 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.042425 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":109,"thread_start_us":106,"threads_started":1}
I20260928 15:05:43.042654 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.031679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.25.123.62:39681)
0928 15:05:43.031892 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.031903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.032047 (+ 144us) server_negotiation.cc:407] Connection header received
0928 15:05:43.032831 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.032833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.032855 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.032970 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.033629 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.034607 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.035636 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.036016 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.036149 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.038778 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.038787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.038787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.038796 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.040740 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.041206 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.041209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.041210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.041302 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.041555 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.041556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.041557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.041842 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.041949 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.042172 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.042366 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260928 15:05:43.043817 7130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.044296 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.044140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49328)
0928 15:05:43.044215 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.044268 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:05:43.044589 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
I20260928 15:05:43.045166 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.045031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49340)
0928 15:05:43.045106 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.045143 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:05:43.045356 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:46633: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.123.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab.
Entry for principal kudu/127.25.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab.
I20260928 15:05:43.068979 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:46633
--webserver_interface=127.25.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab
--principal=kudu/127.25.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39681,127.25.123.61:46633,127.25.123.60:43609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:43.094694 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } attempt: 1
I20260928 15:05:43.095324 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.095135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49346)
0928 15:05:43.095221 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.095283 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:05:43.095837 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.095731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49348)
0928 15:05:43.095782 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.095815 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:05:43.095976 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:46633: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
W20260928 15:05:43.160336 7137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:43.160621 7137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:43.160648 7137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:43.162007 7137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:43.162067 7137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:43.162083 7137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:43.162096 7137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:43.162108 7137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:43.163856 7137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39681,127.25.123.61:46633,127.25.123.60:43609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab
--principal=kudu/127.25.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:46633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:43.164167 7137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:43.164197 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } attempt: 2
I20260928 15:05:43.164525 7137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:05:43.164808 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.164621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49352)
0928 15:05:43.164723 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.164777 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:05:43.165207 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.165113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49364)
0928 15:05:43.165162 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.165189 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 15:05:43.165323 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:46633: Network error: Client connection negotiation failed: client connection to 127.25.123.61:46633: connect: Connection refused (error 111)
W20260928 15:05:43.170288 7145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:43.171061 7142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:43.171121 7143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:43.171528 7137 server_base.cc:1034] running on GCE node
I20260928 15:05:43.171900 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:43.172384 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:43.173691 7137 hybrid_clock.cc:648] HybridClock initialized: now 1790607943173678 us; error 54 us; skew 500 ppm
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:43.177520 7137 init.cc:377] Logged in from keytab as kudu/127.25.123.61@KRBTEST.COM (short username kudu)
I20260928 15:05:43.178514 7137 webserver.cc:460] Webserver started at http://127.25.123.61:41227/ using document root <none> and password file <none>
I20260928 15:05:43.178952 7137 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:43.179085 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:43.179275 7137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:43.180779 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data/instance:
uuid: "0cad2891704745eeb7a0378697897b97"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-qnz0"
server_key: "a97354076fd50c6168dda1c1d9ad2960"
server_key_iv: "9be52bbab30df61db5638a2da01f6357"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.181200 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/wal/instance:
uuid: "0cad2891704745eeb7a0378697897b97"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-qnz0"
server_key: "a97354076fd50c6168dda1c1d9ad2960"
server_key_iv: "9be52bbab30df61db5638a2da01f6357"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.182765 7137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:05:43.183979 7152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:43.184473 7137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:43.184597 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/wal
uuid: "0cad2891704745eeb7a0378697897b97"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-qnz0"
server_key: "a97354076fd50c6168dda1c1d9ad2960"
server_key_iv: "9be52bbab30df61db5638a2da01f6357"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.184670 7137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:43.199453 7137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:43.203744 7137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:43.203903 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:43.208832 7137 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.61:46633
I20260928 15:05:43.209019 7204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.61:46633 every 8 connection(s)
I20260928 15:05:43.209311 7137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
I20260928 15:05:43.210390 7205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:43.212476 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.215260 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7137
I20260928 15:05:43.215759 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-1/wal/instance
I20260928 15:05:43.216039 26092 external_mini_cluster.cc:1428] Setting key 83597e2d45ff264b42f78bebf387034a
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.60@KRBTEST.COM; defaulting to no policy
I20260928 15:05:43.227635 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.213506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.25.123.62:39681)
0928 15:05:43.213643 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.213646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.213872 (+ 226us) server_negotiation.cc:407] Connection header received
0928 15:05:43.214657 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.214660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.214696 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.214773 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.216604 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.217129 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.218327 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.218759 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.218951 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.222740 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.222743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.222744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.222754 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.225184 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.226417 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.226425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.226426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.226515 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.226847 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.226848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.226848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.226993 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.227096 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.227412 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.227530 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 15:05:43.227880 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.213386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39681 (local address 127.0.0.1:43314)
0928 15:05:43.213738 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.213780 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.214322 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.214912 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.214918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.215982 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.216365 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.216374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.217252 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.217253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.218207 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.218211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.219035 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.219037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.219306 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.220174 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.220188 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.222545 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.225422 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.225443 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.225447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.226253 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.226601 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.226602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.226602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.226745 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.227088 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.227095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.227264 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.227391 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.227601 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260928 15:05:43.228941 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.60@KRBTEST.COM" created.
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:05:43.240190 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.229332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49376)
0928 15:05:43.229476 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.229493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.229546 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.229788 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.229789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.229925 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.230025 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.230028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.231308 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.231310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.232152 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.232155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.232991 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.232993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.233129 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.233769 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.233775 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.235515 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.238666 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.238667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.238668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.238930 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.239296 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.239297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.239298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.239372 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.239824 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.239826 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.239895 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.239984 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.240104 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:05:43.240490 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.229382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.25.123.61:46633)
0928 15:05:43.229593 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.229599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.229618 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:05:43.229641 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.229643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.229657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.229737 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.230212 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.231072 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.232260 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.232870 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.232959 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.235708 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.235724 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.235724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.235734 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.238537 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.239051 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.239056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.239057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.239157 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.239457 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.239458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.239459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.239715 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.239820 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.240050 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.240184 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":168,"thread_start_us":130,"threads_started":1}
I20260928 15:05:43.241110 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.241523 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.241390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55564)
0928 15:05:43.241444 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.241499 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 15:05:43.241958 7156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
I20260928 15:05:43.242703 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.242541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55572)
0928 15:05:43.242638 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.242681 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 15:05:43.242890 7205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:43609: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab.
Entry for principal kudu/127.25.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab.
I20260928 15:05:43.255915 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:43609
--webserver_interface=127.25.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab
--principal=kudu/127.25.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:39681,127.25.123.61:46633,127.25.123.60:43609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:43.256507 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } attempt: 3
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:05:43.266191 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.256932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49384)
0928 15:05:43.257076 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.257091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.257160 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.257320 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.257321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.257513 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.257595 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.257598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.258388 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.258389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.258926 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.258928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.259737 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.259738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.259820 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.260468 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.260474 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.262240 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.264972 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.264973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.264974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.265209 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.265454 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.265455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.265456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.265516 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.265877 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.265878 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.265930 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.266001 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.266110 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 15:05:43.266566 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.257013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.25.123.61:46633)
0928 15:05:43.257090 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.257093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.257138 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:05:43.257235 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.257236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.257250 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.257314 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.257691 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.258269 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.259038 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.259472 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.259639 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.262362 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.262365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.262366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.262377 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.264840 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.265298 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.265299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.265300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.265371 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.265588 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.265589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.265590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.265778 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.265859 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.266168 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.266279 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:05:43.267131 7130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.267576 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.267443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55578)
0928 15:05:43.267508 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.267549 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 15:05:43.268049 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.267959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55582)
0928 15:05:43.268006 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.268033 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 15:05:43.268164 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:43609: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
I20260928 15:05:43.292371 7205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } attempt: 1
I20260928 15:05:43.293280 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.293042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55586)
0928 15:05:43.293172 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.293233 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 15:05:43.293901 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.293679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55598)
0928 15:05:43.293796 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.293860 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 15:05:43.294332 7205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:43609: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
I20260928 15:05:43.299404 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } attempt: 1
I20260928 15:05:43.300055 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.299815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55604)
0928 15:05:43.299931 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.300003 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 15:05:43.300663 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.300564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55618)
0928 15:05:43.300607 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.300634 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 15:05:43.300838 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:43609: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
W20260928 15:05:43.358808 7212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:43.359071 7212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:43.359100 7212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:43.361061 7212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:43.361136 7212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:43.361152 7212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:43.361166 7212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:43.361178 7212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:43.362664 7205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } attempt: 2
I20260928 15:05:43.363343 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.363150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55624)
0928 15:05:43.363243 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.363291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 15:05:43.363808 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.363653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55630)
0928 15:05:43.363749 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.363784 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 15:05:43.364097 7205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:43609: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
I20260928 15:05:43.364507 7212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:39681,127.25.123.61:46633,127.25.123.60:43609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab
--principal=kudu/127.25.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:43609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:43.364877 7212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:43.365269 7212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:43.371220 7220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:43.371236 7217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:43.371224 7218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:43.371960 7212 server_base.cc:1034] running on GCE node
I20260928 15:05:43.372375 7212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:43.372773 7212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:43.373996 7212 hybrid_clock.cc:648] HybridClock initialized: now 1790607943373960 us; error 55 us; skew 500 ppm
I20260928 15:05:43.376061 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } attempt: 2
I20260928 15:05:43.376871 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.376541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55646)
0928 15:05:43.376802 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.376839 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:43.377487 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.377345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55662)
0928 15:05:43.377422 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.377463 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 15:05:43.377593 7212 init.cc:377] Logged in from keytab as kudu/127.25.123.60@KRBTEST.COM (short username kudu)
W20260928 15:05:43.377628 7130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:43609: Network error: Client connection negotiation failed: client connection to 127.25.123.60:43609: connect: Connection refused (error 111)
I20260928 15:05:43.378151 7212 webserver.cc:460] Webserver started at http://127.25.123.60:43933/ using document root <none> and password file <none>
I20260928 15:05:43.378405 7212 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:43.378477 7212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:43.378634 7212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:43.379793 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data/instance:
uuid: "b2c10f7682a04ee49d6e95c25c7cfd10"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-qnz0"
server_key: "96765865848089edf4ad9796064ea265"
server_key_iv: "488ff6eefc549f4a6fc133feeb1613bc"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.380185 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/wal/instance:
uuid: "b2c10f7682a04ee49d6e95c25c7cfd10"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-qnz0"
server_key: "96765865848089edf4ad9796064ea265"
server_key_iv: "488ff6eefc549f4a6fc133feeb1613bc"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.381803 7212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:05:43.382890 7227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:43.383297 7212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:43.383410 7212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/wal
uuid: "b2c10f7682a04ee49d6e95c25c7cfd10"
format_stamp: "Formatted at 2026-09-28 15:05:43 on dist-test-slave-qnz0"
server_key: "96765865848089edf4ad9796064ea265"
server_key_iv: "488ff6eefc549f4a6fc133feeb1613bc"
server_key_version: "encryptionkey@0"
I20260928 15:05:43.383522 7212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:43.397701 7212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:43.400913 7212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:43.401048 7212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:43.407096 7212 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.60:43609
I20260928 15:05:43.407133 7279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.60:43609 every 8 connection(s)
I20260928 15:05:43.407547 7212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
I20260928 15:05:43.408625 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:43.410591 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.413904 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7212
I20260928 15:05:43.414045 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790607748072058-26092-0/minicluster-data/master-2/wal/instance
I20260928 15:05:43.414268 26092 external_mini_cluster.cc:1428] Setting key bc5c724faeaaa3c7de87bdbc2c64884f
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:05:43.422832 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.411445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.25.123.62:39681)
0928 15:05:43.411526 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.411529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.411901 (+ 372us) server_negotiation.cc:407] Connection header received
0928 15:05:43.412391 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.412415 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.412427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.412483 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.413255 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.413846 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.415584 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.415897 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.415958 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.418994 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.418996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.418996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.419005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.420850 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.421495 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.421497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.421497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.421571 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.421947 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.421948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.421949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.422141 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.422193 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.422631 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.422732 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:05:43.423413 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.411390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:39681 (local address 127.0.0.1:43322)
0928 15:05:43.411711 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.411753 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.412260 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.412563 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.412567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.412861 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.413076 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.413080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.414133 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.414136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.415475 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.415480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.415992 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.415993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.416179 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.416977 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.416989 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.418912 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.420953 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.420966 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.420968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.421396 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.421659 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.421661 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.421661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.421856 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.422249 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.422257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.422457 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.422922 (+ 465us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.423137 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260928 15:05:43.425288 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.426957 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.414774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.25.123.60:43609)
0928 15:05:43.414982 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.414987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.415013 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:05:43.415046 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.415048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.415066 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.415148 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.415857 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.416946 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.418031 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.418532 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.418601 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.422432 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.422438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.422438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.422444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.424953 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.425637 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.425641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.425641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.425725 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.425978 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.425979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.425980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.426185 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.426332 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.426584 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.426761 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":120,"threads_started":1}
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:43.435807 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.425644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:46633 (local address 127.0.0.1:49388)
0928 15:05:43.425699 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.425710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.425749 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.426265 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.426267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.426434 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.426507 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.426509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.427431 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.427432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.428393 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.428397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.429581 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.429583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.429699 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.430522 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.430546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.432232 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.434411 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.434413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.434413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.434712 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.434953 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.434954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.434955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.435023 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.435412 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.435413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.435467 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.435561 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.435692 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 15:05:43.435798 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.426053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.25.123.61:46633)
0928 15:05:43.426101 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.426103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.426112 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:05:43.426138 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.426139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.426150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.426207 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.426604 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.427323 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.428547 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.428983 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.429295 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.432320 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.432322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.432322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.432331 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.434282 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.434825 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.434826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.434827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.434894 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.435097 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.435099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.435099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.435305 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.435419 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.435598 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.435691 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:05:43.436724 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:05:43.439110 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.428953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.25.123.62:39681)
0928 15:05:43.429063 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.429065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.429073 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:05:43.429099 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.429099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.429111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.429162 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.429963 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.430678 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.431788 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.432097 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.432182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.434895 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.434897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.434897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.434906 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.437218 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.437672 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.437674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.437674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.437769 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.438170 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.438171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.438172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.438319 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.438667 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.438733 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.438980 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:05:43.446823 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.437470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55680)
0928 15:05:43.437530 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.437542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.437583 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.437724 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.437725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.437886 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.437939 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.437941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.439025 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.439028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.439935 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.439938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.440533 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.440536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.440644 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.441278 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.441284 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.442713 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.445502 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.445504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.445504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.445754 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.446002 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.446003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.446004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.446070 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.446456 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.446457 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.446514 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.446597 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.446730 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 15:05:43.446961 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.437510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.25.123.60:43609)
0928 15:05:43.437563 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.437566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.437575 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:05:43.437634 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.437635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.437653 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.437706 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.438103 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.438765 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.440061 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.440364 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.440501 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.442837 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.442839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.442839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.442847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.445324 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.445856 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.445857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.445858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.445919 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.446160 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.446162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.446162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.446354 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.446440 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.446662 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.446857 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":31}
I20260928 15:05:43.449118 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: Bootstrap starting.
I20260928 15:05:43.450110 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.440234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.25.123.61:46633)
0928 15:05:43.440277 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.440279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.440286 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:05:43.440311 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.440311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.440322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.440430 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.441430 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.441959 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.443024 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.443330 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.443408 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.446785 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.446787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.446787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.446798 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.448724 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.449132 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.449134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.449134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.449202 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.449472 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.449473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.449474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.449638 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.449800 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.449959 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.450038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:05:43.450312 7280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:43.450717 7280 log.cc:824] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:43.451756 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: No bootstrap required, opened a new log
I20260928 15:05:43.451982 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:05:43.453888 7280 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.454103 7280 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:43.454150 7280 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c10f7682a04ee49d6e95c25c7cfd10, State: Initialized, Role: FOLLOWER
I20260928 15:05:43.454301 7280 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.454664 7280 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:43.455435 7205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } attempt: 3
I20260928 15:05:43.457866 7212 master_runner.cc:186] Error getting master registration for 127.25.123.62:39681: OK, instance_id { permanent_uuid: "215486b95b3b4b768f75111d000b1878" instance_seqno: 1790607943026060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:05:43.462126 7212 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 15:05:43.457355 7305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:05:43.462224 7305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607943, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:05:43.465425 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.453837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.25.123.60:43609)
0928 15:05:43.453882 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.453884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.454210 (+ 326us) server_negotiation.cc:407] Connection header received
0928 15:05:43.454313 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.454314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.454325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.454358 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.455373 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.456152 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.457438 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.457728 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.457772 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.459935 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.459936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.459937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.459946 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.463064 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.464250 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.464251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.464252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.464313 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.464591 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.464592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.464592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.464789 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.464894 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.465224 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.465311 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:05:43.466020 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.453684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.25.123.62:39681)
0928 15:05:43.454036 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.454038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.454125 (+ 87us) server_negotiation.cc:407] Connection header received
0928 15:05:43.454246 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.454247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.454261 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.454299 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.455374 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.456140 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.457346 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.457657 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.457704 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.459940 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.459942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.459942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.459952 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.462309 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.463586 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.463588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.463589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.463661 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.464785 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.464787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.464787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.464942 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.465064 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.465818 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.465916 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260928 15:05:43.468006 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.455718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55696)
0928 15:05:43.455760 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.455772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.455817 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.458306 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.458307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.458523 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.458598 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.458600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.461164 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.461165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.462103 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.462108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.462815 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.462816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.462886 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.463295 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.463301 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.464877 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.466855 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.466856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.466856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.467057 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.467277 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.467278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.467278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.467339 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.467631 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.467632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.467710 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.467791 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.467903 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 15:05:43.468046 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.456274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.25.123.60:43609)
0928 15:05:43.458132 (+ 1858us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.458134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.458145 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:05:43.458176 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.458176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.458187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.458297 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.460274 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.461037 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.462272 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.462693 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.463050 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.464987 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.464989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.464989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.464996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.466756 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.467128 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.467129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.467129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.467201 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.467432 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.467433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.467433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.467550 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.467627 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.467818 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.467904 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1831,"thread_start_us":93,"threads_started":1}
I20260928 15:05:43.468189 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.453788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.25.123.61:46633)
0928 15:05:43.453838 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.453840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.453989 (+ 149us) server_negotiation.cc:407] Connection header received
0928 15:05:43.454261 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.454262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.454275 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.454342 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.456432 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.457141 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.458313 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.458621 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.458666 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.460787 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.460790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.460791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.460801 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.463969 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.466687 (+ 2718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.466690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.466690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.466760 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.466973 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.466974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.466974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.467113 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.467736 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.467998 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.468087 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:05:43.468562 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.468814 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.470391 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97: Bootstrap starting.
I20260928 15:05:43.471323 7205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:43.471881 7205 log.cc:824] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:43.473556 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97: No bootstrap required, opened a new log
I20260928 15:05:43.475275 7205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.475425 7205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:43.475452 7205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad2891704745eeb7a0378697897b97, State: Initialized, Role: FOLLOWER
I20260928 15:05:43.475607 7205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.475947 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.476034 7205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:43.476039 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.476778 7130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } attempt: 3
W20260928 15:05:43.478699 7319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:05:43.478854 7319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:05:43.478701 7137 master_runner.cc:186] Error getting master registration for 127.25.123.62:39681: OK, instance_id { permanent_uuid: "215486b95b3b4b768f75111d000b1878" instance_seqno: 1790607943026060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:05:43.482029 7137 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 15:05:43 dist-test-slave-qnz0 krb5kdc[7049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607942, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:05:43.487164 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.477236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:43609 (local address 127.0.0.1:55708)
0928 15:05:43.477302 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:43.477316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:43.477369 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:43.477608 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:43.477610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:43.477819 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:43.477874 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.477876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.478857 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.478858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.479760 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:43.479763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.481231 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.481232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:43.481310 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.481885 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:43.481891 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:43.483679 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:43.485815 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.485817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.485818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.486079 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.486322 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:43.486323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:43.486323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:43.486446 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:43.486791 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:43.486792 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:43.486848 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:05:43.486927 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:05:43.487050 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 15:05:43.487648 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:43.477322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.25.123.60:43609)
0928 15:05:43.477459 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:43.477461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:43.477480 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:05:43.477509 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:43.477510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:43.477522 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:43.477575 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:43.477974 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.478745 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.480540 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:43.480986 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:43.481224 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:43.483801 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:43.483803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:43.483803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:43.483813 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:43.485696 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.486169 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.486171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.486171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.486239 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:43.486507 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:43.486508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:43.486533 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:43.486717 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:43.486793 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:43.486969 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 15:05:43.487541 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 15:05:43.489775 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: Bootstrap starting.
I20260928 15:05:43.490695 7130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:43.491037 7130 log.cc:824] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:43.492045 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: No bootstrap required, opened a new log
I20260928 15:05:43.494508 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.494690 7130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:43.494728 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215486b95b3b4b768f75111d000b1878, State: Initialized, Role: FOLLOWER
I20260928 15:05:43.494894 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.495205 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:43.498874 7308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:43.498991 7308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.499253 7308 leader_election.cc:288] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 215486b95b3b4b768f75111d000b1878 (127.25.123.62:39681), b2c10f7682a04ee49d6e95c25c7cfd10 (127.25.123.60:43609)
I20260928 15:05:43.499648 7254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cad2891704745eeb7a0378697897b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" is_pre_election: true
I20260928 15:05:43.499722 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cad2891704745eeb7a0378697897b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215486b95b3b4b768f75111d000b1878" is_pre_election: true
I20260928 15:05:43.499816 7254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cad2891704745eeb7a0378697897b97 in term 0.
I20260928 15:05:43.499845 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cad2891704745eeb7a0378697897b97 in term 0.
I20260928 15:05:43.500034 7156 leader_election.cc:302] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad2891704745eeb7a0378697897b97, 215486b95b3b4b768f75111d000b1878; no voters:
I20260928 15:05:43.500204 7308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:43.500257 7308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:43.500288 7308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:05:43.500857 7333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:05:43.500982 7333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:05:43.501322 7308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.501466 7308 leader_election.cc:288] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [CANDIDATE]: Term 1 election: Requested vote from peers 215486b95b3b4b768f75111d000b1878 (127.25.123.62:39681), b2c10f7682a04ee49d6e95c25c7cfd10 (127.25.123.60:43609)
I20260928 15:05:43.501729 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cad2891704745eeb7a0378697897b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215486b95b3b4b768f75111d000b1878"
I20260928 15:05:43.501811 7104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:43.501791 7254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cad2891704745eeb7a0378697897b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10"
I20260928 15:05:43.501888 7254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:43.502787 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cad2891704745eeb7a0378697897b97 in term 1.
I20260928 15:05:43.502980 7156 leader_election.cc:302] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad2891704745eeb7a0378697897b97, 215486b95b3b4b768f75111d000b1878; no voters:
I20260928 15:05:43.502982 7254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cad2891704745eeb7a0378697897b97 in term 1.
I20260928 15:05:43.504022 7308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:43.504212 7308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [term 1 LEADER]: Becoming Leader. State: Replica: 0cad2891704745eeb7a0378697897b97, State: Running, Role: LEADER
I20260928 15:05:43.504304 7308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } }
I20260928 15:05:43.504740 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cad2891704745eeb7a0378697897b97. Latest consensus state: current_term: 1 leader_uuid: "0cad2891704745eeb7a0378697897b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.504822 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:43.505254 7062 master_runner.cc:263] No leader master found from master 215486b95b3b4b768f75111d000b1878
I20260928 15:05:43.505223 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.505312 7062 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:05:43.505319 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.512667 7337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:43.512970 7337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:43.515621 7254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [term 1 FOLLOWER]: Refusing update from remote peer 0cad2891704745eeb7a0378697897b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:43.516021 7335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:43.517894 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cad2891704745eeb7a0378697897b97. Latest consensus state: current_term: 1 leader_uuid: "0cad2891704745eeb7a0378697897b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.518004 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.518138 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [term 1 FOLLOWER]: Refusing update from remote peer 0cad2891704745eeb7a0378697897b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:43.518410 7335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:43.519882 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cad2891704745eeb7a0378697897b97. Latest consensus state: current_term: 1 leader_uuid: "0cad2891704745eeb7a0378697897b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.519994 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.521416 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cad2891704745eeb7a0378697897b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.521534 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:43.522061 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cad2891704745eeb7a0378697897b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.522150 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.522505 7338 mvcc.cc:205] Tried to move back new op lower bound from 7334330136636645376 to 7334330136594391040. Current Snapshot: MvccSnapshot[applied={T|T < 7334330136636645376}]
I20260928 15:05:43.523749 7341 mvcc.cc:205] Tried to move back new op lower bound from 7334330136636645376 to 7334330136594391040. Current Snapshot: MvccSnapshot[applied={T|T < 7334330136636645376}]
I20260928 15:05:43.525118 7337 catalog_manager.cc:1347] Generated new cluster ID: 5efdfaf155774509bdad659abb2236bf
I20260928 15:05:43.525223 7337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:43.529551 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cad2891704745eeb7a0378697897b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.529831 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:43.530579 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cad2891704745eeb7a0378697897b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215486b95b3b4b768f75111d000b1878" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 39681 } } peers { permanent_uuid: "0cad2891704745eeb7a0378697897b97" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 46633 } } peers { permanent_uuid: "b2c10f7682a04ee49d6e95c25c7cfd10" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 43609 } } }
I20260928 15:05:43.530699 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:05:43.531155 7339 mvcc.cc:205] Tried to move back new op lower bound from 7334330136636645376 to 7334330136594391040. Current Snapshot: MvccSnapshot[applied={T|T < 7334330136636645376}]
I20260928 15:05:43.561452 7337 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:43.562296 7337 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:43.578047 7337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cad2891704745eeb7a0378697897b97: Generated new TSK 0
I20260928 15:05:43.578342 7337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:05:44.462841 7305 catalog_manager.cc:1259] Loaded cluster ID: 5efdfaf155774509bdad659abb2236bf
I20260928 15:05:44.462939 7305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: loading cluster ID for follower catalog manager: success
I20260928 15:05:44.463832 7305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: acquiring CA information for follower catalog manager: success
I20260928 15:05:44.464218 7305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2c10f7682a04ee49d6e95c25c7cfd10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:05:44.501669 7333 catalog_manager.cc:1259] Loaded cluster ID: 5efdfaf155774509bdad659abb2236bf
I20260928 15:05:44.501767 7333 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: loading cluster ID for follower catalog manager: success
I20260928 15:05:44.502719 7333 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: acquiring CA information for follower catalog manager: success
I20260928 15:05:44.503062 7333 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 215486b95b3b4b768f75111d000b1878: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:05:44.639408 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7062
I20260928 15:05:44.644923 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7137
I20260928 15:05:44.650138 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7212
2026-09-28T15:05:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3043 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:05:44 dist-test-slave-qnz0 krb5kdc[7353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:05:44 dist-test-slave-qnz0 krb5kdc[7353](info): set up 2 sockets
Sep 28 15:05:44 dist-test-slave-qnz0 krb5kdc[7353](info): commencing operation
krb5kdc: starting...
W20260928 15:05:45.735846 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:05:45 dist-test-slave-qnz0 krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:05:45.890184 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:43869
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:43869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:45.993285 7366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:45.993554 7366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:45.993582 7366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:45.995028 7366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:45.995257 7366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:45.995340 7366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:45.995360 7366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:45.995375 7366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:46.000497 7366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:43869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:43869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:46.001093 7366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:46.001641 7366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:46.009555 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:46.009903 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:46.010025 7366 server_base.cc:1034] running on GCE node
W20260928 15:05:46.009536 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:46.011315 7366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:46.011919 7366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:46.013931 7366 hybrid_clock.cc:648] HybridClock initialized: now 1790607946013895 us; error 61 us; skew 500 ppm
Sep 28 15:05:46 dist-test-slave-qnz0 krb5kdc[7353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607946, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:46.018131 7366 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:05:46.018735 7366 webserver.cc:460] Webserver started at http://127.25.123.62:42865/ using document root <none> and password file <none>
I20260928 15:05:46.019018 7366 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:46.019069 7366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:46.019351 7366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:46.020797 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "cae4ae4811604ee9b29c5a78acd7d71b"
format_stamp: "Formatted at 2026-09-28 15:05:46 on dist-test-slave-qnz0"
server_key: "47302645c6a2303a90f923ebcea8e9d4"
server_key_iv: "a38f63ae96cbf25d52f599d05d8ccbae"
server_key_version: "encryptionkey@0"
I20260928 15:05:46.021258 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "cae4ae4811604ee9b29c5a78acd7d71b"
format_stamp: "Formatted at 2026-09-28 15:05:46 on dist-test-slave-qnz0"
server_key: "47302645c6a2303a90f923ebcea8e9d4"
server_key_iv: "a38f63ae96cbf25d52f599d05d8ccbae"
server_key_version: "encryptionkey@0"
I20260928 15:05:46.022951 7366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:05:46.024451 7381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:46.024654 7366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:46.024794 7366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "cae4ae4811604ee9b29c5a78acd7d71b"
format_stamp: "Formatted at 2026-09-28 15:05:46 on dist-test-slave-qnz0"
server_key: "47302645c6a2303a90f923ebcea8e9d4"
server_key_iv: "a38f63ae96cbf25d52f599d05d8ccbae"
server_key_version: "encryptionkey@0"
I20260928 15:05:46.024893 7366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:46.035851 7366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:46.040143 7366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:46.040310 7366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:46.045164 7366 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:43869
I20260928 15:05:46.045208 7433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:43869 every 8 connection(s)
I20260928 15:05:46.045732 7366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:46.046967 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7366
I20260928 15:05:46.047080 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:05:46.047231 7434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:46.047322 26092 external_mini_cluster.cc:1428] Setting key 6d1a0c6fec881a10bad309c1e482c3fe
I20260928 15:05:46.052158 7434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b: Bootstrap starting.
I20260928 15:05:46.053388 7434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:46.053730 7434 log.cc:824] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:46.054808 7434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b: No bootstrap required, opened a new log
Sep 28 15:05:46 dist-test-slave-qnz0 krb5kdc[7353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:46.057458 7434 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43869 } }
I20260928 15:05:46.057629 7434 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:46.057652 7434 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae4ae4811604ee9b29c5a78acd7d71b, State: Initialized, Role: FOLLOWER
I20260928 15:05:46.057803 7434 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43869 } }
I20260928 15:05:46.057886 7434 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:46.057912 7434 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:46.057982 7434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:46.058996 7434 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43869 } }
I20260928 15:05:46.059144 7434 leader_election.cc:302] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae4ae4811604ee9b29c5a78acd7d71b; no voters:
I20260928 15:05:46.059376 7434 leader_election.cc:288] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:05:46.059443 7439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:46.059638 7439 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [term 1 LEADER]: Becoming Leader. State: Replica: cae4ae4811604ee9b29c5a78acd7d71b, State: Running, Role: LEADER
I20260928 15:05:46.059736 7434 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:46.059741 7439 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43869 } }
I20260928 15:05:46.060989 7440 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae4ae4811604ee9b29c5a78acd7d71b. Latest consensus state: current_term: 1 leader_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43869 } } }
I20260928 15:05:46.061123 7440 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:46.061005 7441 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae4ae4811604ee9b29c5a78acd7d71b" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43869 } } }
I20260928 15:05:46.061285 7441 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:46.061465 7450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:46.062065 7450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:46.062822 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:46.048129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.25.123.62:43869)
0928 15:05:46.048313 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:46.048321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:46.048348 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:05:46.048887 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:46.048895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:46.048950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:46.049085 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:46.050347 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:46.051608 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:46.053281 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:46.053911 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:46.054037 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:46.057472 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:46.057478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:46.057478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:46.057494 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:46.060205 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:46.061313 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:46.061317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:46.061318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:46.061438 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:46.061730 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:46.061730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:46.061731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:46.062033 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:46.062236 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:46.062316 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 15:05:46.062470 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260928 15:05:46.063879 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:05:46.064159 7450 catalog_manager.cc:1347] Generated new cluster ID: d54e7031b8604ecea52b437be073cd82
I20260928 15:05:46.064224 7450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:46.083583 7450 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:46.084599 7450 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:46.095112 7450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae4ae4811604ee9b29c5a78acd7d71b: Generated new TSK 0
I20260928 15:05:46.095373 7450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:05:46.119076 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7366
2026-09-28T15:05:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1467 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:05:46 dist-test-slave-qnz0 krb5kdc[7461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:05:46 dist-test-slave-qnz0 krb5kdc[7461](info): set up 2 sockets
Sep 28 15:05:46 dist-test-slave-qnz0 krb5kdc[7461](info): commencing operation
krb5kdc: starting...
W20260928 15:05:47.271863 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:05:47 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:05:47.416860 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33915
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:47.507596 7474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:47.507898 7474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:47.507941 7474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:47.509279 7474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:47.509362 7474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:47.509387 7474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:47.509407 7474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:47.509459 7474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:47.512024 7474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:33915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:47.512434 7474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:47.512724 7474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:05:47.519433 7474 server_base.cc:1034] running on GCE node
W20260928 15:05:47.519436 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:47.519433 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:47.519677 7482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:47.519974 7474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:47.520342 7474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:47.521576 7474 hybrid_clock.cc:648] HybridClock initialized: now 1790607947521538 us; error 72 us; skew 500 ppm
Sep 28 15:05:47 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:47.524670 7474 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:05:47.525208 7474 webserver.cc:460] Webserver started at http://127.25.123.62:46055/ using document root <none> and password file <none>
I20260928 15:05:47.525454 7474 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:47.525566 7474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:47.525727 7474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:47.526849 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "40ddc0d736da4f7b834f68674cf6c671"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "4395203d1cf288c1c402d148fc7dc515"
server_key_iv: "7d1cbd24857c464708338a093a166b18"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.527238 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "40ddc0d736da4f7b834f68674cf6c671"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "4395203d1cf288c1c402d148fc7dc515"
server_key_iv: "7d1cbd24857c464708338a093a166b18"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.528808 7474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:05:47.529934 7489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:47.530206 7474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:47.530325 7474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "40ddc0d736da4f7b834f68674cf6c671"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "4395203d1cf288c1c402d148fc7dc515"
server_key_iv: "7d1cbd24857c464708338a093a166b18"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.530412 7474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:47.548542 7474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:47.551167 7474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:47.551362 7474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:47.555997 7474 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:33915
I20260928 15:05:47.556059 7541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:33915 every 8 connection(s)
I20260928 15:05:47.556520 7474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:47.557650 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:47.560135 7542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Bootstrap starting.
I20260928 15:05:47.560811 7542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:47.561113 7542 log.cc:824] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:47.561919 7542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: No bootstrap required, opened a new log
I20260928 15:05:47.563798 7542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:47.563956 7542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:47.563980 7542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ddc0d736da4f7b834f68674cf6c671, State: Initialized, Role: FOLLOWER
I20260928 15:05:47.563997 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7474
I20260928 15:05:47.564179 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:05:47.564146 7542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:47.564245 7542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:47.564288 7542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:47.564359 7542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:47.564384 26092 external_mini_cluster.cc:1428] Setting key 69bf0a1736d8a2ebee28fb62d657ef3f
I20260928 15:05:47.565696 7542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:47.565817 7542 leader_election.cc:302] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ddc0d736da4f7b834f68674cf6c671; no voters:
I20260928 15:05:47.565985 7542 leader_election.cc:288] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:05:47.566035 7547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:47.566191 7547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 LEADER]: Becoming Leader. State: Replica: 40ddc0d736da4f7b834f68674cf6c671, State: Running, Role: LEADER
I20260928 15:05:47.566278 7542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:47.566354 7547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:47.567435 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ddc0d736da4f7b834f68674cf6c671. Latest consensus state: current_term: 1 leader_uuid: "40ddc0d736da4f7b834f68674cf6c671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } } }
I20260928 15:05:47.567543 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:47.567970 7563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:47.568367 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40ddc0d736da4f7b834f68674cf6c671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } } }
I20260928 15:05:47.568558 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:47.568564 7563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:47.570745 7563 catalog_manager.cc:1347] Generated new cluster ID: 51cdbfbccab549debb445beac6eef762
I20260928 15:05:47.570811 7563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:05:47 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:47.577342 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:47.564987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.25.123.62:33915)
0928 15:05:47.565238 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:47.565252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:47.565299 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:05:47.565439 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:47.565446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:47.565505 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:47.565660 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:47.566529 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:47.568087 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:47.569192 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:47.569790 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:47.569912 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:47.573171 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:47.573176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:47.573177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:47.573194 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:47.575469 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:47.575938 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:47.575941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:47.575942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:47.576067 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:47.576290 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:47.576290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:47.576291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:47.576669 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:47.576820 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:47.576956 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 15:05:47.577104 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":118,"threads_started":1}
I20260928 15:05:47.598183 7563 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:05:47.598829 7563 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:47.602030 7563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Generated new TSK 0
I20260928 15:05:47.602305 7563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:05:47.640273 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33915
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:47.752063 7568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:47.752357 7568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:47.752404 7568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:47.753876 7568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:47.753979 7568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:47.754032 7568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:05:47.756124 7568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:47.756702 7568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:47.756994 7568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:47.763554 7574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:47.763751 7573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:47.763554 7576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:47.763921 7568 server_base.cc:1034] running on GCE node
I20260928 15:05:47.764179 7568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:47.764619 7568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:47.765832 7568 hybrid_clock.cc:648] HybridClock initialized: now 1790607947765772 us; error 80 us; skew 500 ppm
Sep 28 15:05:47 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:47.769814 7568 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:05:47.770409 7568 webserver.cc:460] Webserver started at http://127.25.123.1:43777/ using document root <none> and password file <none>
I20260928 15:05:47.770685 7568 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:47.770763 7568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:47.770921 7568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:47.772071 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "8971f32643cf4f478ede41b5e98b3d31"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "e74314e75ad3887a65c7c565f3450461"
server_key_iv: "2c98ca069a4208452e6ee5137caa896f"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.772541 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "8971f32643cf4f478ede41b5e98b3d31"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "e74314e75ad3887a65c7c565f3450461"
server_key_iv: "2c98ca069a4208452e6ee5137caa896f"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.774099 7568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:47.775259 7583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:47.775767 7568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:47.775916 7568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "8971f32643cf4f478ede41b5e98b3d31"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "e74314e75ad3887a65c7c565f3450461"
server_key_iv: "2c98ca069a4208452e6ee5137caa896f"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.776052 7568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:47.788889 7568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:47.792176 7568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:47.792373 7568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:47.792748 7568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:47.793139 7568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:47.793198 7568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:47.793258 7568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:47.793279 7568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:47.801021 7568 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:44389
I20260928 15:05:47.801136 7696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:44389 every 8 connection(s)
I20260928 15:05:47.801553 7568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:47.809706 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7568
I20260928 15:05:47.809885 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:05:47.810187 26092 external_mini_cluster.cc:1428] Setting key cd693ecd70f9a2504fedef4fd96f2e4b
Sep 28 15:05:47 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:47.817751 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:47.803164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:53853 (local address 127.25.123.62:33915)
0928 15:05:47.803245 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:47.803248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:47.803661 (+ 413us) server_negotiation.cc:407] Connection header received
0928 15:05:47.804579 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:47.804581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:47.804608 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:47.804719 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:47.806519 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:47.807115 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:47.808673 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:47.808932 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:47.809044 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:47.813340 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:47.813342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:47.813342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:47.813352 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:47.815911 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:47.816501 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:47.816503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:47.816503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:47.816582 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:47.816873 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:47.816874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:47.816875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:47.817015 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:47.817093 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:47.817416 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 15:05:47.817522 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:05:47.817989 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:47.803213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:53853)
0928 15:05:47.803411 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:47.803458 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:47.804266 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:47.804876 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:47.804880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:47.805786 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:47.806283 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:47.806291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:47.807368 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:47.807370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:47.808527 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:47.808533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:47.809125 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:47.809127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:47.809353 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:47.810396 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:47.810408 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:47.813124 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:47.816077 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:47.816089 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:47.816092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:47.816374 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:47.816655 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:47.816656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:47.816657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:47.816792 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:47.817099 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:47.817107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:47.817265 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 15:05:47.817376 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 15:05:47.817668 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260928 15:05:47.819011 7697 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33915
I20260928 15:05:47.819141 7697 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:47.819437 7697 heartbeater.cc:511] Master 127.25.123.62:33915 requested a full tablet report, sending...
I20260928 15:05:47.820237 7506 ts_manager.cc:194] Registered new tserver with Master: 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1:44389)
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:05:47.821208 7506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:53853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:05:47.845458 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33915
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:47.936601 7702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:47.937011 7702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:47.937042 7702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:47.938901 7702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:47.938969 7702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:47.939004 7702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:05:47.940830 7702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:47.941314 7702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:47.941668 7702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:47.948082 7707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:47.948240 7708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:47.948197 7710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:47.948206 7702 server_base.cc:1034] running on GCE node
I20260928 15:05:47.948647 7702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:47.948994 7702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:47.950214 7702 hybrid_clock.cc:648] HybridClock initialized: now 1790607947950163 us; error 75 us; skew 500 ppm
Sep 28 15:05:47 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:47.953909 7702 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:05:47.954591 7702 webserver.cc:460] Webserver started at http://127.25.123.2:37299/ using document root <none> and password file <none>
I20260928 15:05:47.954922 7702 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:47.954979 7702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:47.955143 7702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:47.957049 7702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "c2e4b45670cf47b38a322d2e90dee241"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "a2b982cdec64a2563932de2e12eb457d"
server_key_iv: "5bb994d48e35378c1c3ee0818664aad9"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.957809 7702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "c2e4b45670cf47b38a322d2e90dee241"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "a2b982cdec64a2563932de2e12eb457d"
server_key_iv: "5bb994d48e35378c1c3ee0818664aad9"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.959553 7702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:05:47.960706 7717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:47.960942 7702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:05:47.961046 7702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "c2e4b45670cf47b38a322d2e90dee241"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "a2b982cdec64a2563932de2e12eb457d"
server_key_iv: "5bb994d48e35378c1c3ee0818664aad9"
server_key_version: "encryptionkey@0"
I20260928 15:05:47.961122 7702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:47.995733 7702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:47.998392 7702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:47.998519 7702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:47.998723 7702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:47.999001 7702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:47.999034 7702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:47.999058 7702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:47.999071 7702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.007994 7702 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:42069
I20260928 15:05:48.008368 7702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:48.008463 7830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:42069 every 8 connection(s)
I20260928 15:05:48.012822 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7702
I20260928 15:05:48.012923 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:05:48.013170 26092 external_mini_cluster.cc:1428] Setting key 8893a8e7c64e887c1318f40438c16f57
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607947, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:48.021835 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.009437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:46911 (local address 127.25.123.62:33915)
0928 15:05:48.009563 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.009565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.009584 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:05:48.010572 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.010574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.010590 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.010692 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.011609 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.012149 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.013620 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.013845 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.013938 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.018156 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.018159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.018160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.018177 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.020070 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.020705 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.020707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.020708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.020776 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.021084 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.021085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.021086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.021221 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.021309 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.021627 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.021732 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 15:05:48.022079 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.009211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:46911)
0928 15:05:48.009390 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.009438 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.010399 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.010835 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.010846 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.011174 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:48.011390 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.011394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.012265 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.012267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.013518 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.013521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.013950 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.013952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.014305 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.015230 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:48.015244 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:48.017832 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:48.020206 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.020220 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.020223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.020612 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.020850 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.020850 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.020851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.020990 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.021293 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:48.021300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:48.021470 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.021584 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.021790 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260928 15:05:48.022722 7831 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33915
I20260928 15:05:48.022840 7831 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:05:48.023126 7831 heartbeater.cc:511] Master 127.25.123.62:33915 requested a full tablet report, sending...
I20260928 15:05:48.023710 7506 ts_manager.cc:194] Registered new tserver with Master: c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069)
I20260928 15:05:48.024081 7506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:46911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:05:48.052726 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33915
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:48.143879 7836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:48.144133 7836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:48.144166 7836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:48.145951 7836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:48.146027 7836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:48.146063 7836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:05:48.147921 7836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:48.148612 7836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:48.149171 7836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:48.155906 7844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.155934 7842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.156126 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:48.156186 7836 server_base.cc:1034] running on GCE node
I20260928 15:05:48.156453 7836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:48.156839 7836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:48.158010 7836 hybrid_clock.cc:648] HybridClock initialized: now 1790607948157986 us; error 36 us; skew 500 ppm
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:48.161065 7836 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:05:48.161638 7836 webserver.cc:460] Webserver started at http://127.25.123.3:45621/ using document root <none> and password file <none>
I20260928 15:05:48.161867 7836 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:48.161914 7836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:48.162019 7836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:05:48.163166 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "2cf19ec9bd684ae888995c74c28733de"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-qnz0"
server_key: "4a418d3663cbdc11cdf58e3598ae3457"
server_key_iv: "40cc5cd7e90ed71fbcc5226bda80cf2d"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.163534 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "2cf19ec9bd684ae888995c74c28733de"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-qnz0"
server_key: "4a418d3663cbdc11cdf58e3598ae3457"
server_key_iv: "40cc5cd7e90ed71fbcc5226bda80cf2d"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.165328 7836 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:05:48.166347 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.166558 7836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:48.166668 7836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "2cf19ec9bd684ae888995c74c28733de"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-qnz0"
server_key: "4a418d3663cbdc11cdf58e3598ae3457"
server_key_iv: "40cc5cd7e90ed71fbcc5226bda80cf2d"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.166756 7836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:48.183679 7836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:48.187006 7836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:48.187271 7836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:48.187537 7836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:48.187870 7836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:48.187920 7836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.187964 7836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:48.187990 7836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.195138 7836 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:42497
I20260928 15:05:48.195242 7964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:42497 every 8 connection(s)
I20260928 15:05:48.195586 7836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:48.201149 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 7836
I20260928 15:05:48.201269 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:05:48.201496 26092 external_mini_cluster.cc:1428] Setting key 606ba71c49e1f63be7dfa41fb2841e7d
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:48.207233 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.196491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:52187 (local address 127.25.123.62:33915)
0928 15:05:48.196555 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.196557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.196780 (+ 223us) server_negotiation.cc:407] Connection header received
0928 15:05:48.197270 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.197271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.197282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.197351 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.198549 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.199078 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.200506 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.200724 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.200792 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.203736 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.203738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.203739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.203746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.205611 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.206097 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.206099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.206099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.206161 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.206505 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.206506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.206506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.206647 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.206712 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.207076 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.207160 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 15:05:48.207432 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.196370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:52187)
0928 15:05:48.196583 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.196630 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.197130 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.197441 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.197445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.198122 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:48.198327 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.198331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.199289 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.199292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.200344 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.200349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.200824 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.200826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.200993 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.201698 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:48.201708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:48.203604 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:48.205745 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.205754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.205756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.206020 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.206225 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.206226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.206226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.206329 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.206760 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:48.206766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:48.206920 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.207022 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.207168 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260928 15:05:48.208357 7965 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33915
I20260928 15:05:48.208544 7965 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:48.208776 7965 heartbeater.cc:511] Master 127.25.123.62:33915 requested a full tablet report, sending...
I20260928 15:05:48.209298 7506 ts_manager.cc:194] Registered new tserver with Master: 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497)
I20260928 15:05:48.209802 7506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:52187
I20260928 15:05:48.214287 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:48.223219 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.215828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.25.123.62:33915)
0928 15:05:48.215906 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.215912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.215925 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:05:48.215961 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.215961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.215973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.216059 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.216816 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.217337 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.218142 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.218334 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.218403 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.219821 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.219823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.219823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.219831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.221701 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.222117 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.222118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.222118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.222180 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.222460 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.222461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.222461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.222674 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.222788 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.223058 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.223145 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 15:05:48.225538 7506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:05:48.227373 7506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:05:48.235368 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.230476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:42497 (local address 127.0.0.1:36900)
0928 15:05:48.230767 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.230817 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.230899 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.231332 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.231336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.231338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:48.231429 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.231431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.232505 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.232506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.233823 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.233826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.234898 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.234899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.234993 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.235078 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.235187 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.235272 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":143,"threads_started":1}
I20260928 15:05:48.235381 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.230805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.25.123.3:42497)
0928 15:05:48.231122 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.231126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.231141 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:05:48.231196 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.231198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.231217 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.231295 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:48.231532 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.232383 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.233924 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.234699 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.234783 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.234842 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.235219 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.235272 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260928 15:05:48.235872 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.230542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44389 (local address 127.0.0.1:47520)
0928 15:05:48.231548 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.231557 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.231628 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.231854 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.231855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.231858 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:48.231896 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.231897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.233180 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.233181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.234230 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.234239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.235589 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.235590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.235661 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.235666 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.235729 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.235778 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":124,"threads_started":1}
I20260928 15:05:48.236217 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.230672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.25.123.1:44389)
0928 15:05:48.231039 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.231045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.231696 (+ 651us) server_negotiation.cc:407] Connection header received
0928 15:05:48.231743 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.231745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.231761 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.231845 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:48.231977 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.233060 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.234467 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.235276 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.235356 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.235410 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.236087 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.236140 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":85,"threads_started":1}
I20260928 15:05:48.236536 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.230476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:42069 (local address 127.0.0.1:47930)
0928 15:05:48.231200 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.231210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.231272 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.231481 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.231482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.231484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:48.231530 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.231532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.232909 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.232910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.234216 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.234219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.236173 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.236174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.236238 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.236242 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.236318 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.236380 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":57,"thread_start_us":146,"threads_started":1}
I20260928 15:05:48.236547 7899 tablet_service.cc:1467] Processing CreateTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 (DEFAULT_TABLE table=test-table [id=c141722d61fc41ef91014436a77ad24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:48.236805 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.230857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.25.123.2:42069)
0928 15:05:48.231138 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.231143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.231334 (+ 191us) server_negotiation.cc:407] Connection header received
0928 15:05:48.231384 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.231386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.231399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.231470 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:48.231627 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.232806 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.234406 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.236000 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.236078 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.236139 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.236666 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.236723 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":90,"threads_started":1}
I20260928 15:05:48.236938 7899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29814800dd8646b6abd76f9cb1aa6d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:48.236904 7631 tablet_service.cc:1467] Processing CreateTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 (DEFAULT_TABLE table=test-table [id=c141722d61fc41ef91014436a77ad24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:48.237185 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29814800dd8646b6abd76f9cb1aa6d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:48.239900 7982 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Bootstrap starting.
I20260928 15:05:48.240774 7982 tablet_bootstrap.cc:654] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:48.240969 7765 tablet_service.cc:1467] Processing CreateTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 (DEFAULT_TABLE table=test-table [id=c141722d61fc41ef91014436a77ad24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:48.241379 7765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29814800dd8646b6abd76f9cb1aa6d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:48.241380 7982 log.cc:824] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:48.242218 7982 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: No bootstrap required, opened a new log
I20260928 15:05:48.242286 7982 ts_tablet_manager.cc:1397] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:05:48.244678 7982 raft_consensus.cc:348] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.244889 7982 raft_consensus.cc:374] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:48.244948 7982 raft_consensus.cc:729] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf19ec9bd684ae888995c74c28733de, State: Initialized, Role: FOLLOWER
I20260928 15:05:48.244915 7983 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap starting.
I20260928 15:05:48.245124 7982 consensus_queue.cc:260] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.245208 7985 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Bootstrap starting.
I20260928 15:05:48.245476 7982 ts_tablet_manager.cc:1428] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 15:05:48.245489 7965 heartbeater.cc:503] Master 127.25.123.62:33915 was elected leader, sending a full tablet report...
I20260928 15:05:48.245780 7983 tablet_bootstrap.cc:654] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:48.246093 7985 tablet_bootstrap.cc:654] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:48.246443 7983 log.cc:824] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:48.246706 7985 log.cc:824] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:48.247426 7983 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: No bootstrap required, opened a new log
I20260928 15:05:48.247429 7985 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: No bootstrap required, opened a new log
I20260928 15:05:48.247516 7983 ts_tablet_manager.cc:1397] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:05:48.247534 7985 ts_tablet_manager.cc:1397] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:05:48.249085 7983 raft_consensus.cc:348] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.249262 7983 raft_consensus.cc:374] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:48.249173 7985 raft_consensus.cc:348] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.249313 7983 raft_consensus.cc:729] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8971f32643cf4f478ede41b5e98b3d31, State: Initialized, Role: FOLLOWER
I20260928 15:05:48.249362 7985 raft_consensus.cc:374] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:48.249442 7985 raft_consensus.cc:729] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e4b45670cf47b38a322d2e90dee241, State: Initialized, Role: FOLLOWER
I20260928 15:05:48.249470 7983 consensus_queue.cc:260] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.249652 7985 consensus_queue.cc:260] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.250064 7985 ts_tablet_manager.cc:1428] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:05:48.250175 7831 heartbeater.cc:503] Master 127.25.123.62:33915 was elected leader, sending a full tablet report...
I20260928 15:05:48.250213 7983 ts_tablet_manager.cc:1428] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:05:48.250804 7697 heartbeater.cc:503] Master 127.25.123.62:33915 was elected leader, sending a full tablet report...
W20260928 15:05:48.259069 7832 tablet.cc:2369] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:05:48.302634 7698 tablet.cc:2369] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:48.330072 7990 raft_consensus.cc:482] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:48.330267 7990 raft_consensus.cc:504] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.330706 7990 leader_election.cc:288] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069), 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497)
I20260928 15:05:48.334421 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.331154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:39847 (local address 127.25.123.2:42069)
0928 15:05:48.331228 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.331230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.331334 (+ 104us) server_negotiation.cc:407] Connection header received
0928 15:05:48.331407 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.331408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.331423 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.331485 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:48.331796 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.332343 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.333524 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.334022 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.334079 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.334087 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.334281 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.334318 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 15:05:48.334419 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.331044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:42069 (local address 127.25.123.1:39847)
0928 15:05:48.331204 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.331214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.331327 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.331524 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.331609 (+ 85us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.331616 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:48.331693 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.331696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.332480 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.332481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.333406 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.333408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.334119 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.334120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.334198 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.334204 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.334272 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.334315 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":37,"thread_start_us":62,"threads_started":1}
I20260928 15:05:48.334903 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.331033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:59939 (local address 127.25.123.3:42497)
0928 15:05:48.331095 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.331097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.331250 (+ 153us) server_negotiation.cc:407] Connection header received
0928 15:05:48.331339 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.331341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.331352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.331506 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:48.331645 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.332209 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.333868 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.334469 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.334556 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.334582 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.334782 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.334825 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 15:05:48.335132 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.330957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:42497 (local address 127.25.123.1:59939)
0928 15:05:48.331150 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.331161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.331256 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.331474 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.331475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.331477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:48.331567 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.331569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.332301 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.332302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.333742 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.333745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.334635 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.334636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.334704 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.334707 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.335007 (+ 300us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.335060 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
I20260928 15:05:48.335152 7785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29814800dd8646b6abd76f9cb1aa6d83" candidate_uuid: "8971f32643cf4f478ede41b5e98b3d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e4b45670cf47b38a322d2e90dee241" is_pre_election: true
I20260928 15:05:48.335554 7785 raft_consensus.cc:2454] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8971f32643cf4f478ede41b5e98b3d31 in term 0.
I20260928 15:05:48.335665 7919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29814800dd8646b6abd76f9cb1aa6d83" candidate_uuid: "8971f32643cf4f478ede41b5e98b3d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf19ec9bd684ae888995c74c28733de" is_pre_election: true
I20260928 15:05:48.335855 7919 raft_consensus.cc:2454] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8971f32643cf4f478ede41b5e98b3d31 in term 0.
I20260928 15:05:48.335896 7587 leader_election.cc:302] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8971f32643cf4f478ede41b5e98b3d31, c2e4b45670cf47b38a322d2e90dee241; no voters:
I20260928 15:05:48.336022 7990 raft_consensus.cc:2781] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:48.336077 7990 raft_consensus.cc:482] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:48.336107 7990 raft_consensus.cc:3037] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:48.336916 7990 raft_consensus.cc:504] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.337085 7990 leader_election.cc:288] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 1 election: Requested vote from peers c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069), 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497)
I20260928 15:05:48.337422 7785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29814800dd8646b6abd76f9cb1aa6d83" candidate_uuid: "8971f32643cf4f478ede41b5e98b3d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e4b45670cf47b38a322d2e90dee241"
I20260928 15:05:48.337445 7919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29814800dd8646b6abd76f9cb1aa6d83" candidate_uuid: "8971f32643cf4f478ede41b5e98b3d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf19ec9bd684ae888995c74c28733de"
I20260928 15:05:48.337551 7919 raft_consensus.cc:3037] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:48.337531 7785 raft_consensus.cc:3037] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:48.338317 7919 raft_consensus.cc:2454] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8971f32643cf4f478ede41b5e98b3d31 in term 1.
I20260928 15:05:48.338328 7785 raft_consensus.cc:2454] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8971f32643cf4f478ede41b5e98b3d31 in term 1.
I20260928 15:05:48.338507 7587 leader_election.cc:302] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8971f32643cf4f478ede41b5e98b3d31, c2e4b45670cf47b38a322d2e90dee241; no voters:
I20260928 15:05:48.338609 7990 raft_consensus.cc:2781] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:48.338797 7990 raft_consensus.cc:686] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 LEADER]: Becoming Leader. State: Replica: 8971f32643cf4f478ede41b5e98b3d31, State: Running, Role: LEADER
I20260928 15:05:48.338907 7990 consensus_queue.cc:237] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.339978 7506 catalog_manager.cc:5495] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1). New cstate: current_term: 1 leader_uuid: "8971f32643cf4f478ede41b5e98b3d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } health_report { overall_health: UNKNOWN } } }
I20260928 15:05:48.354924 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.351837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.25.123.1:44389)
0928 15:05:48.351970 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.351972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.352015 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:05:48.352128 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.352129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.352152 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.352225 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:48.352519 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.353019 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.353830 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.354099 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.354182 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.354458 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:48.354594 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:48.354697 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.354765 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.354788 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 15:05:48.357300 7785 raft_consensus.cc:1261] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Refusing update from remote peer 8971f32643cf4f478ede41b5e98b3d31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:48.357300 7919 raft_consensus.cc:1261] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 1 FOLLOWER]: Refusing update from remote peer 8971f32643cf4f478ede41b5e98b3d31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:48.357843 7990 consensus_queue.cc:1035] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:48.358058 7993 consensus_queue.cc:1035] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:48.360759 7996 mvcc.cc:205] Tried to move back new op lower bound from 7334330156468113408 to 7334330156396871680. Current Snapshot: MvccSnapshot[applied={T|T < 7334330156468113408}]
I20260928 15:05:48.361438 7997 mvcc.cc:205] Tried to move back new op lower bound from 7334330156468113408 to 7334330156396871680. Current Snapshot: MvccSnapshot[applied={T|T < 7334330156468113408}]
I20260928 15:05:48.362370 7998 mvcc.cc:205] Tried to move back new op lower bound from 7334330156468113408 to 7334330156396871680. Current Snapshot: MvccSnapshot[applied={T|T < 7334330156468113408}]
I20260928 15:05:48.363492 7506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:48.363593 7506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:48.364414 7506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Sending DeleteTablet for 3 replicas of tablet 29814800dd8646b6abd76f9cb1aa6d83
I20260928 15:05:48.364791 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:48 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47520
I20260928 15:05:48.364884 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7568
I20260928 15:05:48.364892 7899 tablet_service.cc:1514] Processing DeleteTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:48 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36900
I20260928 15:05:48.364887 7765 tablet_service.cc:1514] Processing DeleteTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:48 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47930
I20260928 15:05:48.370272 8007 tablet_replica.cc:330] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: stopping tablet replica
I20260928 15:05:48.370498 8007 raft_consensus.cc:2229] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:48.370659 8007 raft_consensus.cc:2258] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:48.370873 8006 tablet_replica.cc:330] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: stopping tablet replica
I20260928 15:05:48.371037 8006 raft_consensus.cc:2229] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:48.371184 8006 raft_consensus.cc:2258] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 15:05:48.371325 7491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:44389 (error 108)
I20260928 15:05:48.371738 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7702
I20260928 15:05:48.372711 8007 ts_tablet_manager.cc:1905] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:48.373006 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.371951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44389 (local address 127.0.0.1:47528)
0928 15:05:48.372912 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.372972 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":933}
W20260928 15:05:48.373194 7491 catalog_manager.cc:4584] TS 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1:44389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29814800dd8646b6abd76f9cb1aa6d83: Network error: Client connection negotiation failed: client connection to 127.25.123.1:44389: connect: Connection refused (error 111)
I20260928 15:05:48.375775 8007 ts_tablet_manager.cc:1918] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:48.375906 8007 log.cc:1196] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/29814800dd8646b6abd76f9cb1aa6d83
I20260928 15:05:48.376199 8007 ts_tablet_manager.cc:1939] T 29814800dd8646b6abd76f9cb1aa6d83 P 2cf19ec9bd684ae888995c74c28733de: Deleting consensus metadata
I20260928 15:05:48.376940 7492 catalog_manager.cc:4841] TS 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497): tablet 29814800dd8646b6abd76f9cb1aa6d83 (table test-table [id=c141722d61fc41ef91014436a77ad24f]) successfully deleted
I20260928 15:05:48.378810 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7836
I20260928 15:05:48.378986 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.378787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:42069 (local address 127.0.0.1:47936)
0928 15:05:48.378878 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.378939 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.2:42069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:05:48.384263 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 7474
I20260928 15:05:48.393460 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33915
--webserver_interface=127.25.123.62
--webserver_port=46055
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:48.498906 8009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:48.499178 8009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:48.499207 8009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:48.500655 8009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:48.500720 8009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:48.500736 8009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:48.500751 8009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:48.500763 8009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:48.502537 8009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:33915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:48.502862 8009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:48.503186 8009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:48.525000 8015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.525004 8016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.525004 8018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:48.525987 8009 server_base.cc:1034] running on GCE node
I20260928 15:05:48.526620 8009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:48.527035 8009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:48.528219 8009 hybrid_clock.cc:648] HybridClock initialized: now 1790607948528179 us; error 59 us; skew 500 ppm
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:48.531108 8009 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:05:48.531668 8009 webserver.cc:460] Webserver started at http://127.25.123.62:46055/ using document root <none> and password file <none>
I20260928 15:05:48.531965 8009 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:48.532030 8009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:48.533712 8009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:48.535104 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.535425 8009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:48.535563 8009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "40ddc0d736da4f7b834f68674cf6c671"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "4395203d1cf288c1c402d148fc7dc515"
server_key_iv: "7d1cbd24857c464708338a093a166b18"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.536020 8009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:48.550441 8009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:48.552951 8009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:48.553122 8009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:48.557785 8009 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:33915
I20260928 15:05:48.557817 8077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:33915 every 8 connection(s)
I20260928 15:05:48.558233 8009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:48.560166 8078 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:05:48.560817 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8009
I20260928 15:05:48.560966 8078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Bootstrap starting.
I20260928 15:05:48.561259 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:44389
--local_ip_for_outbound_sockets=127.25.123.1
--tserver_master_addrs=127.25.123.62:33915
--webserver_port=43777
--webserver_interface=127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:48.571827 8078 log.cc:824] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:48.575347 8078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:48.575747 8078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Bootstrap complete.
I20260928 15:05:48.579331 8078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:48.579715 8078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ddc0d736da4f7b834f68674cf6c671, State: Initialized, Role: FOLLOWER
I20260928 15:05:48.579890 8078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:48.579957 8078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:48.580001 8078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:48.580049 8078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:05:48.581595 8078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:48.581733 8078 leader_election.cc:302] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ddc0d736da4f7b834f68674cf6c671; no voters:
I20260928 15:05:48.581921 8078 leader_election.cc:288] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:05:48.582019 8082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:05:48.582190 8078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:48.582257 8082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 LEADER]: Becoming Leader. State: Replica: 40ddc0d736da4f7b834f68674cf6c671, State: Running, Role: LEADER
I20260928 15:05:48.582567 8082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:48.582939 8083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40ddc0d736da4f7b834f68674cf6c671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } } }
I20260928 15:05:48.583042 8083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:48.583324 8084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ddc0d736da4f7b834f68674cf6c671. Latest consensus state: current_term: 2 leader_uuid: "40ddc0d736da4f7b834f68674cf6c671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } } }
I20260928 15:05:48.583411 8084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:48.583514 8091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:48.584678 8091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:48.584798 8091 catalog_manager.cc:1259] Loaded cluster ID: 51cdbfbccab549debb445beac6eef762
I20260928 15:05:48.584834 8091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:05:48.585161 8091 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 15:05:48.589965 8091 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 15:05:48.595167 8091 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:05:48.595503 8091 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Loaded TSK: 0
I20260928 15:05:48.595826 8091 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:05:48.673561 8080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:48.673841 8080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:48.673888 8080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:48.675292 8080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:48.675381 8080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:48.675431 8080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:05:48.677114 8080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:44389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=43777
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:48.677423 8080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:48.677680 8080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:48.684496 8105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.684509 8108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.684496 8106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:48.684876 8080 server_base.cc:1034] running on GCE node
I20260928 15:05:48.685081 8080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:48.685370 8080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:48.686677 8080 hybrid_clock.cc:648] HybridClock initialized: now 1790607948686638 us; error 36 us; skew 500 ppm
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:48.691089 8080 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:05:48.692101 8080 webserver.cc:460] Webserver started at http://127.25.123.1:43777/ using document root <none> and password file <none>
I20260928 15:05:48.692577 8080 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:48.692667 8080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:48.696202 8080 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:05:48.697213 8115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.697387 8080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:05:48.697489 8080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "8971f32643cf4f478ede41b5e98b3d31"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "e74314e75ad3887a65c7c565f3450461"
server_key_iv: "2c98ca069a4208452e6ee5137caa896f"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.697899 8080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:48.737897 8080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:48.740726 8080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:48.740926 8080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:48.741225 8080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:48.741925 8122 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:05:48.743534 8080 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:05:48.743593 8080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:05:48.743624 8080 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:05:48.744325 8080 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:05:48.744374 8080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:05:48.744443 8122 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap starting.
I20260928 15:05:48.754711 8080 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:44389
I20260928 15:05:48.754809 8229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:44389 every 8 connection(s)
I20260928 15:05:48.755330 8080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:48.758437 8122 log.cc:824] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:48.759001 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8080
I20260928 15:05:48.759332 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:42069
--local_ip_for_outbound_sockets=127.25.123.2
--tserver_master_addrs=127.25.123.62:33915
--webserver_port=37299
--webserver_interface=127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:48.761003 8122 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:48.761335 8122 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap complete.
I20260928 15:05:48.762490 8122 ts_tablet_manager.cc:1397] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260928 15:05:48.764482 8122 raft_consensus.cc:348] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.764914 8122 raft_consensus.cc:729] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8971f32643cf4f478ede41b5e98b3d31, State: Initialized, Role: FOLLOWER
I20260928 15:05:48.765082 8122 consensus_queue.cc:260] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.765453 8122 ts_tablet_manager.cc:1428] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:48.769956 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.756181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:40391)
0928 15:05:48.756376 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.756463 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.757294 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.757851 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.757855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.758304 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:48.758585 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.758592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.762186 (+ 3594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.762188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.763163 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.763167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.763818 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.763820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.763965 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.764559 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:48.764568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:48.766119 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:48.768194 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.768203 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.768205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.768506 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.768790 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.768790 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.768791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.768901 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.769301 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:48.769307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:48.769444 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.769544 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.769741 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260928 15:05:48.769955 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.756343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:40391 (local address 127.25.123.62:33915)
0928 15:05:48.756527 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.756537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.756579 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:05:48.757553 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.757558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.757648 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.757769 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.760557 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.762043 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.763285 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.763702 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.763837 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.766250 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.766255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.766255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.766274 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.768071 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.768607 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.768610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.768611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.768712 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.768973 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.768974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.768975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.769222 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.769310 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.769615 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.769750 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260928 15:05:48.770682 8230 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33915
I20260928 15:05:48.770812 8230 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:48.771042 8230 heartbeater.cc:511] Master 127.25.123.62:33915 requested a full tablet report, sending...
I20260928 15:05:48.771891 8042 ts_manager.cc:194] Registered new tserver with Master: 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1:44389)
I20260928 15:05:48.773320 8042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:40391
I20260928 15:05:48.774282 8230 heartbeater.cc:503] Master 127.25.123.62:33915 was elected leader, sending a full tablet report...
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:05:48.783152 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.772891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.25.123.1:44389)
0928 15:05:48.773049 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.773054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.773064 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:05:48.773099 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.773101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.773116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.773197 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.773792 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.774950 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.775867 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.776383 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.776505 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.779281 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.779292 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.779292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.779300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.781267 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.781957 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.781961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.781961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.782028 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.782354 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.782356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.782356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.782586 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.782678 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.782925 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.783045 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260928 15:05:48.783190 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.772587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44389 (local address 127.0.0.1:47532)
0928 15:05:48.772803 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.772845 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.772921 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.773221 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.773225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.773638 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:48.773705 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.773708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.775039 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.775040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.775663 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.775666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.776661 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.776663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.776737 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.777390 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:48.777396 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:48.779076 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:48.781472 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.781484 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.781487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.781820 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.782143 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.782144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.782145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.782266 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.782688 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:48.782697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:48.782763 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.782860 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.783043 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":123,"threads_started":1}
I20260928 15:05:48.783803 8164 tablet_service.cc:1514] Processing DeleteTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:48 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47532
I20260928 15:05:48.784518 8239 tablet_replica.cc:330] stopping tablet replica
I20260928 15:05:48.784634 8239 raft_consensus.cc:2229] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:48.784708 8239 raft_consensus.cc:2258] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:48.785017 8239 ts_tablet_manager.cc:1905] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:48.787417 8239 ts_tablet_manager.cc:1918] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:48.787503 8239 log.cc:1196] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/29814800dd8646b6abd76f9cb1aa6d83
I20260928 15:05:48.787799 8239 ts_tablet_manager.cc:1939] T 29814800dd8646b6abd76f9cb1aa6d83 P 8971f32643cf4f478ede41b5e98b3d31: Deleting consensus metadata
I20260928 15:05:48.788537 8027 catalog_manager.cc:4841] TS 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1:44389): tablet 29814800dd8646b6abd76f9cb1aa6d83 (table test-table [id=c141722d61fc41ef91014436a77ad24f]) successfully deleted
W20260928 15:05:48.858927 8235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:48.859223 8235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:48.859277 8235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:48.860680 8235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:48.860759 8235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:48.860813 8235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:05:48.862507 8235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:42069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=37299
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:48.862834 8235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:48.863117 8235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:48.868829 8244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.868822 8247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:48.868822 8245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:48.869336 8235 server_base.cc:1034] running on GCE node
I20260928 15:05:48.869614 8235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:48.870146 8235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:48.871430 8235 hybrid_clock.cc:648] HybridClock initialized: now 1790607948871363 us; error 112 us; skew 500 ppm
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:48.874924 8235 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:05:48.875460 8235 webserver.cc:460] Webserver started at http://127.25.123.2:37299/ using document root <none> and password file <none>
I20260928 15:05:48.875720 8235 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:48.875828 8235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:48.877566 8235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:05:48.878607 8254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:48.878988 8235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:05:48.879110 8235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "c2e4b45670cf47b38a322d2e90dee241"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "a2b982cdec64a2563932de2e12eb457d"
server_key_iv: "5bb994d48e35378c1c3ee0818664aad9"
server_key_version: "encryptionkey@0"
I20260928 15:05:48.879496 8235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:48.893312 8235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:48.895843 8235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:48.896045 8235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:48.896307 8235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:48.896828 8261 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:05:48.898512 8235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:05:48.898660 8235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:05:48.898736 8235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:05:48.899511 8235 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:05:48.899581 8235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:05:48.899706 8261 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Bootstrap starting.
I20260928 15:05:48.908453 8235 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:42069
I20260928 15:05:48.908515 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:42069 every 8 connection(s)
I20260928 15:05:48.908905 8235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:48.912796 8261 log.cc:824] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:48.916149 8261 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:48.916591 8261 tablet_bootstrap.cc:492] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Bootstrap complete.
I20260928 15:05:48.916635 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8235
I20260928 15:05:48.916901 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:42497
--local_ip_for_outbound_sockets=127.25.123.3
--tserver_master_addrs=127.25.123.62:33915
--webserver_port=45621
--webserver_interface=127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:48.917945 8261 ts_tablet_manager.cc:1397] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.007s
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:48.919468 8261 raft_consensus.cc:348] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.919909 8261 raft_consensus.cc:729] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e4b45670cf47b38a322d2e90dee241, State: Initialized, Role: FOLLOWER
I20260928 15:05:48.920149 8261 consensus_queue.cc:260] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } } peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } }
I20260928 15:05:48.920603 8261 ts_tablet_manager.cc:1428] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 15:05:48.923998 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.909750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:38709 (local address 127.25.123.62:33915)
0928 15:05:48.909830 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.909833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.910158 (+ 325us) server_negotiation.cc:407] Connection header received
0928 15:05:48.911684 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.911686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.911705 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.911773 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.913159 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.913890 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.915785 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.916534 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.916688 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.920263 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.920266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.920267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.920278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.922326 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.922846 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.922848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.922848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.922923 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.923225 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.923226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.923227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.923376 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.923458 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.923775 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.923894 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:05:48.924202 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.909683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:38709)
0928 15:05:48.909894 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.909975 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.911085 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.911943 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.911955 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.912603 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:48.912939 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.912949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.914093 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.914095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.915554 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.915558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.916934 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.916936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.917370 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.918098 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:48.918115 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:48.920115 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:48.922456 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.922464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.922467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.922737 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.923014 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.923015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.923016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.923149 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.923449 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:48.923456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:48.923621 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.923739 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.923942 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":108,"threads_started":1}
I20260928 15:05:48.924783 8369 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33915
I20260928 15:05:48.924890 8369 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:48.925168 8369 heartbeater.cc:511] Master 127.25.123.62:33915 requested a full tablet report, sending...
I20260928 15:05:48.925817 8042 ts_manager.cc:194] Registered new tserver with Master: c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069)
I20260928 15:05:48.926448 8042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:38709
I20260928 15:05:48.927496 8369 heartbeater.cc:503] Master 127.25.123.62:33915 was elected leader, sending a full tablet report...
Sep 28 15:05:48 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607948, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.2@KRBTEST.COM
I20260928 15:05:48.936975 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.926309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.25.123.2:42069)
0928 15:05:48.926539 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:48.926546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:48.926558 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:05:48.926601 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:48.926604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:48.926626 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:48.926706 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:48.927026 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.928442 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.929285 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.929819 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.929903 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.932088 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:48.932098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:48.932098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:48.932105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:48.935077 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.935720 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.935726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.935726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.935921 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:48.936214 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:48.936216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:48.936216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:48.936528 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:48.936617 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:48.936796 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 15:05:48.936901 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260928 15:05:48.936996 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:48.926245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:42069 (local address 127.0.0.1:47946)
0928 15:05:48.926314 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:48.926327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:48.926375 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:48.926719 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:48.926720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:48.926897 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:48.926940 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.926942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.928625 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.928626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.929196 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:48.929199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:48.929919 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:48.929920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:48.929977 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:48.930504 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:48.930515 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:48.931966 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:48.935246 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.935247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.935248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.935552 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.936021 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:48.936022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:48.936022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:48.936105 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:48.936610 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:48.936611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:48.936671 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 15:05:48.936749 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:05:48.936880 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:05:48.937461 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 29814800dd8646b6abd76f9cb1aa6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:48 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47946
I20260928 15:05:48.938082 8376 tablet_replica.cc:330] stopping tablet replica
I20260928 15:05:48.938218 8376 raft_consensus.cc:2229] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:48.938285 8376 raft_consensus.cc:2258] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:48.938702 8376 ts_tablet_manager.cc:1905] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:48.941270 8376 ts_tablet_manager.cc:1918] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:48.941375 8376 log.cc:1196] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/29814800dd8646b6abd76f9cb1aa6d83
I20260928 15:05:48.941612 8376 ts_tablet_manager.cc:1939] T 29814800dd8646b6abd76f9cb1aa6d83 P c2e4b45670cf47b38a322d2e90dee241: Deleting consensus metadata
I20260928 15:05:48.942482 8029 catalog_manager.cc:4841] TS c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069): tablet 29814800dd8646b6abd76f9cb1aa6d83 (table test-table [id=c141722d61fc41ef91014436a77ad24f]) successfully deleted
W20260928 15:05:49.015765 8373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:49.016019 8373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:49.016048 8373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:49.017537 8373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:49.017616 8373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:49.017652 8373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:05:49.019359 8373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:42497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=45621
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:49.019649 8373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:49.019918 8373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:49.025979 8382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.025979 8381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.025982 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:49.026391 8373 server_base.cc:1034] running on GCE node
I20260928 15:05:49.026628 8373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:49.027043 8373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:49.028294 8373 hybrid_clock.cc:648] HybridClock initialized: now 1790607949028230 us; error 92 us; skew 500 ppm
Sep 28 15:05:49 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:49.031440 8373 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:05:49.031961 8373 webserver.cc:460] Webserver started at http://127.25.123.3:45621/ using document root <none> and password file <none>
I20260928 15:05:49.032164 8373 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:49.032241 8373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:49.033864 8373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.034679 8391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.035038 8373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.035169 8373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "2cf19ec9bd684ae888995c74c28733de"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-qnz0"
server_key: "4a418d3663cbdc11cdf58e3598ae3457"
server_key_iv: "40cc5cd7e90ed71fbcc5226bda80cf2d"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.035645 8373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:49.060302 8373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:49.063242 8373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:49.063376 8373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:49.063572 8373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:49.063899 8373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:49.063934 8373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.063958 8373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:49.063971 8373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.071206 8373 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:42497
I20260928 15:05:49.071287 8504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:42497 every 8 connection(s)
I20260928 15:05:49.071684 8373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:49.074494 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8373
Sep 28 15:05:49 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:05:49.082537 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.074915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.25.123.62:33915)
0928 15:05:49.075062 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.075064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.075151 (+ 87us) server_negotiation.cc:407] Connection header received
0928 15:05:49.075218 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.075218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.075230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.075288 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:49.076069 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.076837 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.077816 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.078120 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.078202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.079491 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:49.079492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:49.079493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:49.079500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:49.081249 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.081685 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.081686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.081686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.081747 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.081957 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.081958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.081958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.082096 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:49.082256 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.082320 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.082416 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260928 15:05:49.083700 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.072617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:51635 (local address 127.25.123.62:33915)
0928 15:05:49.072709 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.072712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.072918 (+ 206us) server_negotiation.cc:407] Connection header received
0928 15:05:49.073628 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.073629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.073643 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.073743 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:49.075089 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.075999 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.077313 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.077685 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.077773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.080472 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:49.080473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:49.080474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:49.080481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:49.082067 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.082554 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.082555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.082555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.082617 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:49.082916 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:49.082917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:49.082918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:49.083026 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:49.083104 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.083496 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.083561 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:05:49.083948 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.072625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:51635)
0928 15:05:49.072796 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.072834 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.073494 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.073833 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:49.073837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:49.074614 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:49.074956 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.074964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.076102 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.076103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.077230 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.077233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.077831 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.077832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.077989 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.078621 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:49.078630 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:49.080332 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:49.082205 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:49.082213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:49.082215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:49.082486 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:49.082699 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:49.082700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:49.082700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:49.082862 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:49.083131 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:49.083137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:49.083336 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 15:05:49.083456 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 15:05:49.083677 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260928 15:05:49.084569 8505 heartbeater.cc:347] Connected to a master server at 127.25.123.62:33915
I20260928 15:05:49.084669 8505 heartbeater.cc:464] Registering TS with master...
I20260928 15:05:49.084849 8505 heartbeater.cc:511] Master 127.25.123.62:33915 requested a full tablet report, sending...
I20260928 15:05:49.085235 8042 ts_manager.cc:194] Registered new tserver with Master: 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497)
I20260928 15:05:49.085633 8042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:51635
I20260928 15:05:49.095129 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:05:49.098436 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.095569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.25.123.62:33915)
0928 15:05:49.095691 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.095694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.095803 (+ 109us) server_negotiation.cc:407] Connection header received
0928 15:05:49.095907 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.095907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.095955 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.096067 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:49.096237 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.096805 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.097617 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.097810 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.097894 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.098090 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:49.098206 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:49.098301 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.098320 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.098341 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:05:49.098783 8042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:05:49.100535 8042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:05:49.103750 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 006eb2dee26744a488d679ba31be12d0 (DEFAULT_TABLE table=test-table [id=9c8bd521eb97407fa755d24ffbbc2a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:49.103854 8164 tablet_service.cc:1467] Processing CreateTablet for tablet 006eb2dee26744a488d679ba31be12d0 (DEFAULT_TABLE table=test-table [id=9c8bd521eb97407fa755d24ffbbc2a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:49.103992 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006eb2dee26744a488d679ba31be12d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:49.104100 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006eb2dee26744a488d679ba31be12d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:49.105528 8261 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Bootstrap starting.
I20260928 15:05:49.105929 8261 tablet_bootstrap.cc:654] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:49.106119 8122 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap starting.
I20260928 15:05:49.106452 8122 tablet_bootstrap.cc:654] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:49.106622 8261 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: No bootstrap required, opened a new log
I20260928 15:05:49.106668 8261 ts_tablet_manager.cc:1397] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 15:05:49.107044 8122 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: No bootstrap required, opened a new log
I20260928 15:05:49.107016 8261 raft_consensus.cc:348] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.107081 8122 ts_tablet_manager.cc:1397] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.107106 8261 raft_consensus.cc:374] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:49.107156 8261 raft_consensus.cc:729] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e4b45670cf47b38a322d2e90dee241, State: Initialized, Role: FOLLOWER
I20260928 15:05:49.107223 8261 consensus_queue.cc:260] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.107332 8261 ts_tablet_manager.cc:1428] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 15:05:49.107429 8122 raft_consensus.cc:348] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.107508 8122 raft_consensus.cc:374] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:49.107564 8122 raft_consensus.cc:729] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8971f32643cf4f478ede41b5e98b3d31, State: Initialized, Role: FOLLOWER
I20260928 15:05:49.107617 8122 consensus_queue.cc:260] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.107710 8122 ts_tablet_manager.cc:1428] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.109275 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.103656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.25.123.3:42497)
0928 15:05:49.104534 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.104540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.104552 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:05:49.104597 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.104599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.104617 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.104746 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:49.104927 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.106256 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.107955 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.108848 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.108953 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.109024 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.109150 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.109191 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":253,"threads_started":1}
I20260928 15:05:49.109266 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.103461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:42497 (local address 127.0.0.1:36914)
0928 15:05:49.103670 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.103683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.103733 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.104782 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:49.104783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:49.104785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:49.104838 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.104840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.106400 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.106402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.107715 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.107720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.108956 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.108957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.109038 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.109060 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:05:49.109132 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:05:49.109191 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260928 15:05:49.110141 8439 tablet_service.cc:1467] Processing CreateTablet for tablet 006eb2dee26744a488d679ba31be12d0 (DEFAULT_TABLE table=test-table [id=9c8bd521eb97407fa755d24ffbbc2a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:05:49.110495 8439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006eb2dee26744a488d679ba31be12d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:05:49.113606 8517 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Bootstrap starting.
I20260928 15:05:49.114290 8517 tablet_bootstrap.cc:654] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Neither blocks nor log segments found. Creating new log.
I20260928 15:05:49.114871 8517 log.cc:824] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:49.115701 8517 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: No bootstrap required, opened a new log
I20260928 15:05:49.115782 8517 ts_tablet_manager.cc:1397] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:05:49.117375 8517 raft_consensus.cc:348] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.117558 8517 raft_consensus.cc:374] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:05:49.117609 8517 raft_consensus.cc:729] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf19ec9bd684ae888995c74c28733de, State: Initialized, Role: FOLLOWER
I20260928 15:05:49.117746 8517 consensus_queue.cc:260] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.118053 8517 ts_tablet_manager.cc:1428] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:05:49.118124 8505 heartbeater.cc:503] Master 127.25.123.62:33915 was elected leader, sending a full tablet report...
W20260928 15:05:49.159614 8370 tablet.cc:2369] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:05:49.187418 8519 raft_consensus.cc:482] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:49.187778 8519 raft_consensus.cc:504] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.188182 8519 leader_election.cc:288] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069), 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1:44389)
I20260928 15:05:49.191536 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.188434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44389 (local address 127.25.123.3:42263)
0928 15:05:49.188502 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.188515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.188563 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.188870 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:49.188871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:49.188873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:49.188955 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.188957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.189807 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.189808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.190611 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.190613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.191226 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.191227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.191305 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.191309 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:05:49.191367 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 15:05:49.191431 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":18}
I20260928 15:05:49.191556 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.188571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:35725 (local address 127.25.123.2:42069)
0928 15:05:49.188613 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.188615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.188805 (+ 190us) server_negotiation.cc:407] Connection header received
0928 15:05:49.188862 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.188863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.188888 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.188969 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:49.189195 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.189724 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.190708 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.191148 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.191245 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.191311 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.191443 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.191489 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:05:49.191525 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.188526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:42263 (local address 127.25.123.1:44389)
0928 15:05:49.188577 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.188579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.188590 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:05:49.188725 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.188727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.188755 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.188824 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:05:49.189031 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.189702 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.190717 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.191141 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.191202 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.191295 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.191383 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.191441 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:05:49.191622 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.188437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:42069 (local address 127.25.123.3:35725)
0928 15:05:49.188689 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.188701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.188769 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.188978 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:49.188979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:49.188980 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:05:49.189069 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.189071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.189853 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.189853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.190615 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:49.190617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.191262 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.191262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:49.191361 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.191366 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:05:49.191425 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 15:05:49.191460 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":65,"thread_start_us":107,"threads_started":1}
I20260928 15:05:49.191908 8184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006eb2dee26744a488d679ba31be12d0" candidate_uuid: "2cf19ec9bd684ae888995c74c28733de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8971f32643cf4f478ede41b5e98b3d31" is_pre_election: true
I20260928 15:05:49.191962 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006eb2dee26744a488d679ba31be12d0" candidate_uuid: "2cf19ec9bd684ae888995c74c28733de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e4b45670cf47b38a322d2e90dee241" is_pre_election: true
I20260928 15:05:49.192072 8184 raft_consensus.cc:2454] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf19ec9bd684ae888995c74c28733de in term 0.
I20260928 15:05:49.192147 8323 raft_consensus.cc:2454] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf19ec9bd684ae888995c74c28733de in term 0.
I20260928 15:05:49.192353 8393 leader_election.cc:302] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf19ec9bd684ae888995c74c28733de, 8971f32643cf4f478ede41b5e98b3d31; no voters:
I20260928 15:05:49.192530 8519 raft_consensus.cc:2781] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:05:49.192601 8519 raft_consensus.cc:482] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:05:49.192642 8519 raft_consensus.cc:3037] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:49.193519 8519 raft_consensus.cc:504] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.193694 8519 leader_election.cc:288] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [CANDIDATE]: Term 1 election: Requested vote from peers c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069), 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1:44389)
I20260928 15:05:49.193863 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006eb2dee26744a488d679ba31be12d0" candidate_uuid: "2cf19ec9bd684ae888995c74c28733de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e4b45670cf47b38a322d2e90dee241"
I20260928 15:05:49.193953 8323 raft_consensus.cc:3037] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:49.193971 8184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006eb2dee26744a488d679ba31be12d0" candidate_uuid: "2cf19ec9bd684ae888995c74c28733de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8971f32643cf4f478ede41b5e98b3d31"
I20260928 15:05:49.194056 8184 raft_consensus.cc:3037] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:05:49.194818 8323 raft_consensus.cc:2454] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf19ec9bd684ae888995c74c28733de in term 1.
I20260928 15:05:49.194909 8184 raft_consensus.cc:2454] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf19ec9bd684ae888995c74c28733de in term 1.
I20260928 15:05:49.195031 8395 leader_election.cc:302] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf19ec9bd684ae888995c74c28733de, c2e4b45670cf47b38a322d2e90dee241; no voters:
I20260928 15:05:49.195154 8519 raft_consensus.cc:2781] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:05:49.195333 8519 raft_consensus.cc:686] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 1 LEADER]: Becoming Leader. State: Replica: 2cf19ec9bd684ae888995c74c28733de, State: Running, Role: LEADER
I20260928 15:05:49.195442 8519 consensus_queue.cc:237] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.196450 8042 catalog_manager.cc:5495] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cf19ec9bd684ae888995c74c28733de (127.25.123.3). New cstate: current_term: 1 leader_uuid: "2cf19ec9bd684ae888995c74c28733de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } health_report { overall_health: UNKNOWN } } }
I20260928 15:05:49.208659 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.205748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.25.123.3:42497)
0928 15:05:49.205813 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:49.205815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:49.205823 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:05:49.205864 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:49.205865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:49.205890 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:49.205964 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:05:49.206098 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.206578 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.207409 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:49.207620 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:49.207913 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:49.208099 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:05:49.208231 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:05:49.208487 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:49.208510 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 15:05:49.208538 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:05:49.211047 8323 raft_consensus.cc:1261] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Refusing update from remote peer 2cf19ec9bd684ae888995c74c28733de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:49.211047 8184 raft_consensus.cc:1261] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Refusing update from remote peer 2cf19ec9bd684ae888995c74c28733de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:05:49.211410 8521 consensus_queue.cc:1035] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:49.211555 8519 consensus_queue.cc:1035] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:05:49.213801 8525 mvcc.cc:205] Tried to move back new op lower bound from 7334330159965360128 to 7334330159905148928. Current Snapshot: MvccSnapshot[applied={T|T < 7334330159965360128}]
I20260928 15:05:49.214428 8526 mvcc.cc:205] Tried to move back new op lower bound from 7334330159965360128 to 7334330159905148928. Current Snapshot: MvccSnapshot[applied={T|T < 7334330159965360128}]
I20260928 15:05:49.216677 8040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:49.216781 8040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:05:49.217558 8040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Sending DeleteTablet for 3 replicas of tablet 006eb2dee26744a488d679ba31be12d0
I20260928 15:05:49.217859 8439 tablet_service.cc:1514] Processing DeleteTablet for tablet 006eb2dee26744a488d679ba31be12d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:49 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36914
I20260928 15:05:49.217921 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 006eb2dee26744a488d679ba31be12d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:49 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47946
I20260928 15:05:49.217924 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 8080
I20260928 15:05:49.218021 8164 tablet_service.cc:1514] Processing DeleteTablet for tablet 006eb2dee26744a488d679ba31be12d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:05:49 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47532
I20260928 15:05:49.218112 8376 tablet_replica.cc:330] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: stopping tablet replica
I20260928 15:05:49.218161 8376 raft_consensus.cc:2229] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:05:49.218220 8376 raft_consensus.cc:2258] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:49.218750 8534 tablet_replica.cc:330] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: stopping tablet replica
I20260928 15:05:49.218851 8534 raft_consensus.cc:2229] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:05:49.218986 8534 raft_consensus.cc:2258] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:05:49.219197 8376 ts_tablet_manager.cc:1905] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:49.220990 8376 ts_tablet_manager.cc:1918] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:49.221065 8376 log.cc:1196] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/006eb2dee26744a488d679ba31be12d0
I20260928 15:05:49.221302 8376 ts_tablet_manager.cc:1939] T 006eb2dee26744a488d679ba31be12d0 P c2e4b45670cf47b38a322d2e90dee241: Deleting consensus metadata
I20260928 15:05:49.221879 8029 catalog_manager.cc:4841] TS c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069): tablet 006eb2dee26744a488d679ba31be12d0 (table test-table [id=9c8bd521eb97407fa755d24ffbbc2a3e]) successfully deleted
W20260928 15:05:49.223436 8027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:44389 (error 108)
W20260928 15:05:49.224301 8237 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:05:49.223827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:44389 (local address 127.0.0.1:47540)
0928 15:05:49.223894 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.223907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.223961 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.224249 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:44389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 15:05:49.224517 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 8235
W20260928 15:05:49.224462 8027 catalog_manager.cc:4584] TS 8971f32643cf4f478ede41b5e98b3d31 (127.25.123.1:44389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 006eb2dee26744a488d679ba31be12d0: Network error: Client connection negotiation failed: client connection to 127.25.123.1:44389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:05:49.226167 8534 ts_tablet_manager.cc:1905] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:05:49.229120 8534 ts_tablet_manager.cc:1918] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:05:49.229241 8534 log.cc:1196] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/006eb2dee26744a488d679ba31be12d0
I20260928 15:05:49.229529 8534 ts_tablet_manager.cc:1939] T 006eb2dee26744a488d679ba31be12d0 P 2cf19ec9bd684ae888995c74c28733de: Deleting consensus metadata
I20260928 15:05:49.230650 8028 catalog_manager.cc:4841] TS 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497): tablet 006eb2dee26744a488d679ba31be12d0 (table test-table [id=9c8bd521eb97407fa755d24ffbbc2a3e]) successfully deleted
I20260928 15:05:49.232805 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 8373
I20260928 15:05:49.238387 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 8009
I20260928 15:05:49.244484 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33915
--webserver_interface=127.25.123.62
--webserver_port=46055
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:49.334209 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:49.334452 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:49.334479 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:49.335803 8536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:05:49.335863 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:49.335878 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:49.335893 8536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:05:49.335909 8536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:05:49.337618 8536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:33915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:49.337921 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:49.338184 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:49.343969 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.343961 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.343961 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:49.344169 8536 server_base.cc:1034] running on GCE node
I20260928 15:05:49.344543 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:49.344916 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:49.346107 8536 hybrid_clock.cc:648] HybridClock initialized: now 1790607949346069 us; error 47 us; skew 500 ppm
Sep 28 15:05:49 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:49.349249 8536 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:05:49.349864 8536 webserver.cc:460] Webserver started at http://127.25.123.62:46055/ using document root <none> and password file <none>
I20260928 15:05:49.350142 8536 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:49.350214 8536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:49.351718 8536 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.352806 8551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.353098 8536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.353226 8536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "40ddc0d736da4f7b834f68674cf6c671"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "4395203d1cf288c1c402d148fc7dc515"
server_key_iv: "7d1cbd24857c464708338a093a166b18"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.353628 8536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:49.366308 8536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:49.369068 8536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:49.369247 8536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:49.373822 8536 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:33915
I20260928 15:05:49.373864 8603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:33915 every 8 connection(s)
I20260928 15:05:49.374261 8536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:05:49.376585 8604 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:05:49.377367 8604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Bootstrap starting.
I20260928 15:05:49.380700 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8536
I20260928 15:05:49.381049 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:44389
--local_ip_for_outbound_sockets=127.25.123.1
--tserver_master_addrs=127.25.123.62:33915
--webserver_port=43777
--webserver_interface=127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:49.386868 8604 log.cc:824] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:49.389401 8604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:49.389732 8604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671: Bootstrap complete.
I20260928 15:05:49.392050 8604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:49.392315 8604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ddc0d736da4f7b834f68674cf6c671, State: Initialized, Role: FOLLOWER
I20260928 15:05:49.392545 8604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:49.392643 8604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:05:49.392696 8604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:05:49.392748 8604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:05:49.393801 8604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:49.393924 8604 leader_election.cc:302] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ddc0d736da4f7b834f68674cf6c671; no voters:
I20260928 15:05:49.394140 8604 leader_election.cc:288] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:05:49.394181 8608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:05:49.394300 8608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [term 3 LEADER]: Becoming Leader. State: Replica: 40ddc0d736da4f7b834f68674cf6c671, State: Running, Role: LEADER
I20260928 15:05:49.394388 8608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } }
I20260928 15:05:49.394424 8604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:05:49.394775 8609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "40ddc0d736da4f7b834f68674cf6c671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } } }
I20260928 15:05:49.394863 8609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:49.395092 8610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ddc0d736da4f7b834f68674cf6c671. Latest consensus state: current_term: 3 leader_uuid: "40ddc0d736da4f7b834f68674cf6c671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddc0d736da4f7b834f68674cf6c671" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 33915 } } }
I20260928 15:05:49.395215 8610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddc0d736da4f7b834f68674cf6c671 [sys.catalog]: This master's current role is: LEADER
I20260928 15:05:49.395365 8614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:05:49.396235 8614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:05:49.396449 8614 catalog_manager.cc:1259] Loaded cluster ID: 51cdbfbccab549debb445beac6eef762
I20260928 15:05:49.396641 8614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 15:05:49.397087 8614 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f138f9199ad google::LogMessage::Fail() at ??:0
@ 0x7f138f91c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f138f9194de google::LogMessage::Flush() at ??:0
@ 0x7f138f91aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f139783a0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f13978438b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f13917e70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f13917dc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f13926406db start_thread at ??:0
@ 0x7f138e41761f clone at ??:0
W20260928 15:05:49.485875 8606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:49.486155 8606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:49.486203 8606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:49.487771 8606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:49.487869 8606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:49.487923 8606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:05:49.491312 8606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:44389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=43777
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:49.491871 8606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:49.492159 8606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:49.497844 8643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.497857 8640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.497983 8641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:49.498343 8606 server_base.cc:1034] running on GCE node
I20260928 15:05:49.498569 8606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:49.498890 8606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:49.500090 8606 hybrid_clock.cc:648] HybridClock initialized: now 1790607949500080 us; error 32 us; skew 500 ppm
Sep 28 15:05:49 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:49.502899 8606 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:05:49.503455 8606 webserver.cc:460] Webserver started at http://127.25.123.1:43777/ using document root <none> and password file <none>
I20260928 15:05:49.503703 8606 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:49.503772 8606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:49.505528 8606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.508018 8650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.510989 8606 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 15:05:49.511157 8606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "8971f32643cf4f478ede41b5e98b3d31"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "e74314e75ad3887a65c7c565f3450461"
server_key_iv: "2c98ca069a4208452e6ee5137caa896f"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.511746 8606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:49.545857 8606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:49.548362 8606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:49.548576 8606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:49.548833 8606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:49.549299 8657 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:05:49.551606 8606 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:05:49.551704 8606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 15:05:49.551756 8606 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:05:49.552567 8606 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:05:49.552623 8606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.552709 8657 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap starting.
I20260928 15:05:49.561658 8606 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:44389
I20260928 15:05:49.562071 8606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:05:49.562685 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:44389 every 8 connection(s)
I20260928 15:05:49.565471 8657 log.cc:824] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Log is configured to *not* fsync() on all Append() calls
I20260928 15:05:49.567898 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8606
I20260928 15:05:49.568295 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:42069
--local_ip_for_outbound_sockets=127.25.123.2
--tserver_master_addrs=127.25.123.62:33915
--webserver_port=37299
--webserver_interface=127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:05:49.568842 8657 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:05:49.569232 8657 tablet_bootstrap.cc:492] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Bootstrap complete.
I20260928 15:05:49.570660 8657 ts_tablet_manager.cc:1397] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
I20260928 15:05:49.573195 8657 raft_consensus.cc:348] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.573642 8657 raft_consensus.cc:729] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8971f32643cf4f478ede41b5e98b3d31, State: Initialized, Role: FOLLOWER
I20260928 15:05:49.573840 8657 consensus_queue.cc:260] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:49.574182 8657 ts_tablet_manager.cc:1428] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31: Time spent starting tablet: real 0.003s user 0.005s sys 0.001s
W20260928 15:05:49.631175 8767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:05:49.562817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:37719)
0928 15:05:49.562999 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.563048 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:49.563566 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:49.630935 (+ 67369us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
W20260928 15:05:49.631462 8765 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:33915 (0 consecutive failures): Network error: Failed to ping master at 127.25.123.62:33915: Client connection negotiation failed: client connection to 127.25.123.62:33915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:05:49.631844 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.631717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:33261)
0928 15:05:49.631775 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.631821 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 15:05:49.632249 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.632156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:32921)
0928 15:05:49.632200 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.632228 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 15:05:49.632414 8765 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:05:49.656035 8769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:49.656272 8769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:49.656299 8769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:49.658200 8769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:49.658270 8769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:49.658303 8769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:05:49.659919 8769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:42069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=37299
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:49.660148 8769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:49.660351 8769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:05:49.666705 8769 server_base.cc:1034] running on GCE node
W20260928 15:05:49.666541 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.666574 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.666541 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:49.667209 8769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:49.667496 8769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:49.668672 8769 hybrid_clock.cc:648] HybridClock initialized: now 1790607949668655 us; error 59 us; skew 500 ppm
Sep 28 15:05:49 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:49.671646 8769 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:05:49.672195 8769 webserver.cc:460] Webserver started at http://127.25.123.2:37299/ using document root <none> and password file <none>
I20260928 15:05:49.672497 8769 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:49.672549 8769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:49.674141 8769 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.674973 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.675266 8769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.675357 8769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "c2e4b45670cf47b38a322d2e90dee241"
format_stamp: "Formatted at 2026-09-28 15:05:47 on dist-test-slave-qnz0"
server_key: "a2b982cdec64a2563932de2e12eb457d"
server_key_iv: "5bb994d48e35378c1c3ee0818664aad9"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.675755 8769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:49.687353 8769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:49.690063 8769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:49.690265 8769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:49.690573 8769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:49.690937 8769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:49.690976 8769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.691002 8769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:49.691017 8769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.698230 8769 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:42069
I20260928 15:05:49.698290 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:42069 every 8 connection(s)
I20260928 15:05:49.698736 8769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:05:49.700120 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.699612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:47039)
0928 15:05:49.699801 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.699891 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
W20260928 15:05:49.700579 8899 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:33915 (0 consecutive failures): Network error: Failed to ping master at 127.25.123.62:33915: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
I20260928 15:05:49.700954 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.700814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:39223)
0928 15:05:49.700895 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.700930 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 15:05:49.701315 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.701241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:36935)
0928 15:05:49.701266 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.701288 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 15:05:49.701467 8899 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:05:49.704341 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8769
I20260928 15:05:49.704703 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:42497
--local_ip_for_outbound_sockets=127.25.123.3
--tserver_master_addrs=127.25.123.62:33915
--webserver_port=45621
--webserver_interface=127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:05:49.796227 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:05:49.796576 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:05:49.796622 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:05:49.797981 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:05:49.798032 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:05:49.798065 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:05:49.799723 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:42497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=45621
--tserver_master_addrs=127.25.123.62:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:05:49.799961 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:05:49.800231 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:05:49.805814 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.805843 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:05:49.805810 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:05:49.806272 8902 server_base.cc:1034] running on GCE node
I20260928 15:05:49.806561 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:05:49.806836 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:05:49.808050 8902 hybrid_clock.cc:648] HybridClock initialized: now 1790607949808029 us; error 38 us; skew 500 ppm
Sep 28 15:05:49 dist-test-slave-qnz0 krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:05:49.811192 8902 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:05:49.811695 8902 webserver.cc:460] Webserver started at http://127.25.123.3:45621/ using document root <none> and password file <none>
I20260928 15:05:49.811944 8902 fs_manager.cc:359] Metadata directory not provided
I20260928 15:05:49.812014 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:05:49.813498 8902 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.814225 8917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.814428 8902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:05:49.814586 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "2cf19ec9bd684ae888995c74c28733de"
format_stamp: "Formatted at 2026-09-28 15:05:48 on dist-test-slave-qnz0"
server_key: "4a418d3663cbdc11cdf58e3598ae3457"
server_key_iv: "40cc5cd7e90ed71fbcc5226bda80cf2d"
server_key_version: "encryptionkey@0"
I20260928 15:05:49.815214 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:05:49.832741 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:05:49.835172 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:05:49.835366 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:05:49.835642 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:05:49.835981 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:05:49.836038 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.836076 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:05:49.836110 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:05:49.842823 8902 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:42497
I20260928 15:05:49.842897 9031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:42497 every 8 connection(s)
I20260928 15:05:49.843317 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:05:49.844803 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.844329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:52641)
0928 15:05:49.844535 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.844622 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
W20260928 15:05:49.845265 9032 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:33915 (0 consecutive failures): Network error: Failed to ping master at 127.25.123.62:33915: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
I20260928 15:05:49.845618 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.845520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:58635)
0928 15:05:49.845570 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.845599 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 15:05:49.845947 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:49.845893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:49251)
0928 15:05:49.845914 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:49.845934 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 15:05:49.846064 9032 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:05:49.850778 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 8902
I20260928 15:05:50.633464 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.633009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:39385)
0928 15:05:50.633300 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.633386 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":102,"threads_started":1}
I20260928 15:05:50.702308 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.701986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:55715)
0928 15:05:50.702191 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.702240 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260928 15:05:50.847086 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:50.846603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:53439)
0928 15:05:50.846845 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:50.846903 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260928 15:05:51.004184 9039 raft_consensus.cc:482] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:51.004375 9039 raft_consensus.cc:504] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:51.005031 9039 leader_election.cc:288] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497), c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069)
Sep 28 15:05:51 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.3@KRBTEST.COM
Sep 28 15:05:51 dist-test-slave-qnz0 krb5kdc[7461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607949, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.2@KRBTEST.COM
I20260928 15:05:51.019484 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:51.005377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:40575 (local address 127.25.123.2:42069)
0928 15:05:51.005575 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:51.005588 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:51.005625 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:05:51.006163 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:51.006166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:51.006210 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:51.006377 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:51.007384 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.008773 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.010466 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.011005 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.011303 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:51.014018 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:51.014024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:51.014025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:51.014043 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:51.017059 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:51.017759 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:51.017764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:51.017765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:51.017915 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:51.018332 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:51.018333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:51.018334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:51.018811 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:51.018909 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:51.019214 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 15:05:51.019354 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260928 15:05:51.019471 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:51.005301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:42497 (local address 127.25.123.1:53297)
0928 15:05:51.005545 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:51.005558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:51.005639 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:51.006437 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:51.006440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:51.006850 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:51.007079 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:51.007086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.008831 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.008832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:51.009992 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:51.009995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.010946 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.010948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:51.011222 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:51.011798 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:51.011809 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:51.013514 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:51.017211 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:51.017213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:51.017213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:51.017820 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:51.018223 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:51.018224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:51.018224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:51.018326 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:51.018880 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:51.018888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:51.019031 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 15:05:51.019150 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:05:51.019368 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":28,"thread_start_us":110,"threads_started":1}
I20260928 15:05:51.019595 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:51.005387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:53297 (local address 127.25.123.3:42497)
0928 15:05:51.005542 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 15:05:51.005553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:05:51.005653 (+ 100us) server_negotiation.cc:407] Connection header received
0928 15:05:51.006156 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:05:51.006159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:05:51.006210 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:05:51.006396 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:05:51.007316 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.008671 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.010157 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.010840 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.010975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:51.013774 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:05:51.013782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:05:51.013783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:05:51.013803 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:05:51.017060 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:51.017927 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:51.017930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:51.017930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:51.018094 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:05:51.018424 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:05:51.018425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:05:51.018426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:05:51.018802 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:05:51.018909 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:05:51.019215 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 15:05:51.019473 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260928 15:05:51.019999 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:51.005246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:42069 (local address 127.25.123.1:40575)
0928 15:05:51.005312 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:51.005325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:05:51.005383 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:05:51.006437 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:05:51.006440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:05:51.006826 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:05:51.007154 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:51.007158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.008909 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.008910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:51.010308 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:05:51.010311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:05:51.011136 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:05:51.011138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:05:51.011581 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:05:51.012019 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:05:51.012024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:05:51.013867 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:05:51.017183 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:51.017196 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:51.017198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:51.017617 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:51.018046 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:05:51.018047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:05:51.018047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:05:51.018251 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:05:51.018902 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:05:51.018903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:05:51.019032 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 15:05:51.019151 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:05:51.019530 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":254}
W20260928 15:05:51.020964 8653 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
W20260928 15:05:51.021050 8654 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
I20260928 15:05:51.021162 8654 leader_election.cc:302] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8971f32643cf4f478ede41b5e98b3d31; no voters: 2cf19ec9bd684ae888995c74c28733de, c2e4b45670cf47b38a322d2e90dee241
I20260928 15:05:51.021497 9039 raft_consensus.cc:2726] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:05:51.634661 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:51.634377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:33579)
0928 15:05:51.634562 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:51.634611 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260928 15:05:51.703089 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:51.702864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:54139)
0928 15:05:51.703003 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:51.703040 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260928 15:05:51.848043 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:51.847783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:47365)
0928 15:05:51.847950 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:51.847994 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260928 15:05:52.594123 9046 raft_consensus.cc:482] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:52.594239 9046 raft_consensus.cc:504] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:52.594436 9046 leader_election.cc:288] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497), c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069)
W20260928 15:05:52.595043 8654 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
W20260928 15:05:52.595136 8653 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
I20260928 15:05:52.595254 8653 leader_election.cc:302] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8971f32643cf4f478ede41b5e98b3d31; no voters: 2cf19ec9bd684ae888995c74c28733de, c2e4b45670cf47b38a322d2e90dee241
I20260928 15:05:52.595367 9046 raft_consensus.cc:2726] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:05:52.635598 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.635299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:60741)
0928 15:05:52.635470 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:52.635540 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260928 15:05:52.704180 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.703817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:47621)
0928 15:05:52.704053 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:52.704111 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":130,"threads_started":1}
I20260928 15:05:52.849211 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:52.848812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:56645)
0928 15:05:52.849009 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:52.849063 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260928 15:05:53.636632 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.636250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:51955)
0928 15:05:53.636512 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.636578 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":83,"threads_started":1}
I20260928 15:05:53.705309 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.704955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:47743)
0928 15:05:53.705181 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.705253 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":168,"threads_started":1}
I20260928 15:05:53.850291 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:53.849988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:60793)
0928 15:05:53.850170 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:53.850238 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260928 15:05:54.637616 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:54.637347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:40417)
0928 15:05:54.637515 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:54.637564 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260928 15:05:54.706202 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:54.705912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:51069)
0928 15:05:54.706072 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:54.706139 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260928 15:05:54.851253 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:54.851000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:58115)
0928 15:05:54.851148 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:54.851199 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260928 15:05:54.929783 9056 raft_consensus.cc:482] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:54.929951 9056 raft_consensus.cc:504] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:54.930505 9056 leader_election.cc:288] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497), c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069)
W20260928 15:05:54.930907 8654 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
W20260928 15:05:54.930970 8653 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
I20260928 15:05:54.931070 8653 leader_election.cc:302] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8971f32643cf4f478ede41b5e98b3d31; no voters: 2cf19ec9bd684ae888995c74c28733de, c2e4b45670cf47b38a322d2e90dee241
I20260928 15:05:54.931174 9056 raft_consensus.cc:2726] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:05:55.638571 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.638251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:42799)
0928 15:05:55.638489 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.638521 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":132,"threads_started":1}
I20260928 15:05:55.707283 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.706949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:37247)
0928 15:05:55.707182 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.707233 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":147,"threads_started":1}
I20260928 15:05:55.852357 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:55.852051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:36023)
0928 15:05:55.852238 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:55.852294 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260928 15:05:56.639535 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:56.639183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:42377)
0928 15:05:56.639407 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:56.639483 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":164,"threads_started":1}
I20260928 15:05:56.708571 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:56.708186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:51085)
0928 15:05:56.708371 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:56.708487 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260928 15:05:56.853463 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:56.853108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:50671)
0928 15:05:56.853291 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:56.853338 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260928 15:05:57.186812 9063 raft_consensus.cc:482] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:05:57.186923 9063 raft_consensus.cc:504] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:05:57.187309 9063 leader_election.cc:288] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497), c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069)
W20260928 15:05:57.187816 8654 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
W20260928 15:05:57.187882 8653 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
I20260928 15:05:57.187927 8653 leader_election.cc:302] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8971f32643cf4f478ede41b5e98b3d31; no voters: 2cf19ec9bd684ae888995c74c28733de, c2e4b45670cf47b38a322d2e90dee241
I20260928 15:05:57.187999 9063 raft_consensus.cc:2726] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:05:57.640621 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:57.640283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:42633)
0928 15:05:57.640509 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:57.640572 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260928 15:05:57.709573 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:57.709306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:50411)
0928 15:05:57.709480 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:57.709524 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260928 15:05:57.854380 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:57.854125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:56575)
0928 15:05:57.854283 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:57.854331 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260928 15:05:58.641537 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:58.641277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:54415)
0928 15:05:58.641434 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:58.641487 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260928 15:05:58.710811 9068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:58.710386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:51795)
0928 15:05:58.710601 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:58.710757 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260928 15:05:58.855633 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:58.855173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:56809)
0928 15:05:58.855469 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:58.855579 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":235,"threads_started":1}
I20260928 15:05:59.642424 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:59.642153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:42749)
0928 15:05:59.642330 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:59.642373 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260928 15:05:59.712015 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:59.711634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:33907)
0928 15:05:59.711892 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:59.711961 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":106,"threads_started":1}
I20260928 15:05:59.856596 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:05:59.856263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:55361)
0928 15:05:59.856497 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 15:05:59.856539 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":73,"threads_started":1}
I20260928 15:06:00.643347 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:00.643094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:59303)
0928 15:06:00.643244 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:00.643293 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260928 15:06:00.713042 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:00.712751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:34697)
0928 15:06:00.712943 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:00.712989 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260928 15:06:00.857687 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:00.857337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:39375)
0928 15:06:00.857577 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:00.857624 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260928 15:06:01.609316 9076 raft_consensus.cc:482] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:06:01.609462 9076 raft_consensus.cc:504] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf19ec9bd684ae888995c74c28733de" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 42497 } } peers { permanent_uuid: "c2e4b45670cf47b38a322d2e90dee241" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 42069 } } peers { permanent_uuid: "8971f32643cf4f478ede41b5e98b3d31" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 44389 } }
I20260928 15:06:01.609687 9076 leader_election.cc:288] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497), c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069)
W20260928 15:06:01.610169 8653 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cf19ec9bd684ae888995c74c28733de (127.25.123.3:42497): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
W20260928 15:06:01.610249 8654 leader_election.cc:341] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2e4b45670cf47b38a322d2e90dee241 (127.25.123.2:42069): Not found: Tablet not found: 006eb2dee26744a488d679ba31be12d0
I20260928 15:06:01.610317 8654 leader_election.cc:302] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8971f32643cf4f478ede41b5e98b3d31; no voters: 2cf19ec9bd684ae888995c74c28733de, c2e4b45670cf47b38a322d2e90dee241
I20260928 15:06:01.610409 9076 raft_consensus.cc:2726] T 006eb2dee26744a488d679ba31be12d0 P 8971f32643cf4f478ede41b5e98b3d31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:06:01.644631 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:01.644193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:49201)
0928 15:06:01.644528 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:01.644575 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":257,"threads_started":1}
I20260928 15:06:01.714309 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:01.713920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:58025)
0928 15:06:01.714110 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:01.714169 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
I20260928 15:06:01.858793 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:01.858427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:35171)
0928 15:06:01.858688 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:01.858737 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20260928 15:06:02.645624 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:02.645276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:53137)
0928 15:06:02.645477 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:02.645572 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260928 15:06:02.715229 9081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:02.714963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:56297)
0928 15:06:02.715131 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:02.715179 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260928 15:06:02.859750 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:02.859462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:46345)
0928 15:06:02.859653 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:02.859699 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260928 15:06:03.646680 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:03.646332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:48245)
0928 15:06:03.646566 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:03.646623 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260928 15:06:03.716677 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:03.716105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:36063)
0928 15:06:03.716551 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:03.716604 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":365,"threads_started":1}
I20260928 15:06:03.861027 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:03.860595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:34507)
0928 15:06:03.860904 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:03.860961 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":258,"threads_started":1}
I20260928 15:06:04.647617 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:04.647373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.1:41823)
0928 15:06:04.647526 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:04.647571 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260928 15:06:04.717792 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:04.717451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.2:39279)
0928 15:06:04.717692 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:04.717736 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":69,"threads_started":1}
I20260928 15:06:04.862337 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:04.862009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:33915 (local address 127.25.123.3:36445)
0928 15:06:04.862233 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:04.862277 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260928 15:06:04.862344 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 8606
I20260928 15:06:04.868183 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 8769
I20260928 15:06:04.875562 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 8902
2026-09-28T15:06:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18758 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:06:04 dist-test-slave-qnz0 krb5kdc[9091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:06:04 dist-test-slave-qnz0 krb5kdc[9091](info): set up 2 sockets
Sep 28 15:06:04 dist-test-slave-qnz0 krb5kdc[9091](info): commencing operation
krb5kdc: starting...
W20260928 15:06:06.015638 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:06:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:06:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:06:06.165563 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37565
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:37565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:06.261618 9104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:06.261868 9104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:06.261896 9104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:06.263268 9104 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:06:06.263322 9104 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:06:06.263336 9104 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:06:06.263348 9104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:06:06.263362 9104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:06.263376 9104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:06.263424 9104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:06:06.263443 9104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:06:06.265372 9104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:37565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:06.265905 9104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:06.266217 9104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:06:06.272837 9104 server_base.cc:1034] running on GCE node
W20260928 15:06:06.272787 9109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:06.272792 9110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:06.272787 9112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:06.273532 9104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:06.273957 9104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:06.275439 9104 hybrid_clock.cc:648] HybridClock initialized: now 1790607966275415 us; error 38 us; skew 500 ppm
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:06.278771 9104 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:06:06.279374 9104 webserver.cc:460] Webserver started at http://127.25.123.62:36787/ using document root <none> and password file <none>
I20260928 15:06:06.279649 9104 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:06.279699 9104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:06.279870 9104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:06.281095 9104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "a5c510f895966c3c6528d3b805d0ee63"
server_key_iv: "19e83b5082378ca0c178d27b8b91c75d"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.281504 9104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "a5c510f895966c3c6528d3b805d0ee63"
server_key_iv: "19e83b5082378ca0c178d27b8b91c75d"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.283056 9104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:06:06.284148 9119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.284461 9104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:06.284590 9104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "a5c510f895966c3c6528d3b805d0ee63"
server_key_iv: "19e83b5082378ca0c178d27b8b91c75d"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.284695 9104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:06.301404 9104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:06.304579 9104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:06.304863 9104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:06.310456 9104 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:37565
I20260928 15:06:06.310484 9172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:37565 every 8 connection(s)
I20260928 15:06:06.310963 9104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:06:06.311918 9173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:06.312117 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 9104
I20260928 15:06:06.312572 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:06:06.312809 26092 external_mini_cluster.cc:1428] Setting key 8fef3ad2bfbc46164f02f9922ffac449
I20260928 15:06:06.315745 9173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Bootstrap starting.
I20260928 15:06:06.316592 9173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:06.316910 9173 log.cc:824] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:06.318064 9173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: No bootstrap required, opened a new log
I20260928 15:06:06.320953 9173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:06.321237 9173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:06.321391 9173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea32c79a3d84c7991bcb2a7193cfcc5, State: Initialized, Role: FOLLOWER
I20260928 15:06:06.321660 9173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:06.321782 9173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:06:06.321835 9173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:06:06.321893 9173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:06:06.322880 9173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:06.323040 9173 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ea32c79a3d84c7991bcb2a7193cfcc5; no voters:
I20260928 15:06:06.323309 9173 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:06:06.323370 9178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:06:06.323599 9178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 LEADER]: Becoming Leader. State: Replica: 9ea32c79a3d84c7991bcb2a7193cfcc5, State: Running, Role: LEADER
I20260928 15:06:06.323861 9173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:06:06.323768 9178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:06.325139 9180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea32c79a3d84c7991bcb2a7193cfcc5. Latest consensus state: current_term: 1 leader_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } } }
I20260928 15:06:06.325330 9180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:06.325609 9179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } } }
I20260928 15:06:06.326858 9179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: This master's current role is: LEADER
W20260928 15:06:06.327075 9193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:06:06.327152 9193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:06:06.327263 9187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:06:06.327466 9187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:06:06.328827 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.313659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.25.123.62:37565)
0928 15:06:06.313891 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.313904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.313945 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:06:06.314601 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.314610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.314677 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.314895 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:06.315731 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.317155 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.318310 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.318913 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.319070 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.323517 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:06.323524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:06.323524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:06.323542 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:06.326529 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.327148 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.327153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.327153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.327289 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.327604 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.327605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.327605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.328007 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:06.328219 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.328374 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.328575 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260928 15:06:06.329545 9187 catalog_manager.cc:1347] Generated new cluster ID: 445b375ede744372ac883e5657e03b5e
I20260928 15:06:06.329600 9187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:06:06.346944 9187 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:06:06.347817 9187 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:06:06.366452 9187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Generated new TSK 0
I20260928 15:06:06.366703 9187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:06:06.396196 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37565
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:06.491130 9199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:06.491429 9199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:06.491478 9199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:06.492954 9199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:06.493033 9199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:06.493084 9199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:06:06.494808 9199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:06.495119 9199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:06.495393 9199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:06.501428 9205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:06.501428 9204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:06.501451 9207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:06.501853 9199 server_base.cc:1034] running on GCE node
I20260928 15:06:06.502256 9199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:06.502763 9199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:06.503947 9199 hybrid_clock.cc:648] HybridClock initialized: now 1790607966503927 us; error 43 us; skew 500 ppm
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:06.507050 9199 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:06:06.507609 9199 webserver.cc:460] Webserver started at http://127.25.123.1:43961/ using document root <none> and password file <none>
I20260928 15:06:06.507910 9199 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:06.507972 9199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:06.508117 9199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:06.509486 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "30d3bc139e744aa8867d2956b2ec418c"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "0166ebd19b8ee35fcc3b051ff770a801"
server_key_iv: "f38f0e70c53d4c43e4087d9697711ec6"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.509899 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "30d3bc139e744aa8867d2956b2ec418c"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "0166ebd19b8ee35fcc3b051ff770a801"
server_key_iv: "f38f0e70c53d4c43e4087d9697711ec6"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.511417 9199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:06:06.512586 9214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.512845 9199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:06:06.512975 9199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "30d3bc139e744aa8867d2956b2ec418c"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "0166ebd19b8ee35fcc3b051ff770a801"
server_key_iv: "f38f0e70c53d4c43e4087d9697711ec6"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.513077 9199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:06.544793 9199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:06.547402 9199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:06.547818 9199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:06.548149 9199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:06.548542 9199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:06.548614 9199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.548657 9199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:06.548713 9199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.555442 9199 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:46619
I20260928 15:06:06.555649 9328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:46619 every 8 connection(s)
I20260928 15:06:06.555979 9199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:06:06.563304 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 9199
I20260928 15:06:06.563447 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:06:06.563735 26092 external_mini_cluster.cc:1428] Setting key 2b4cc1fbb1a4c975e6112f35dd5a822b
I20260928 15:06:06.567414 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.557274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:57923 (local address 127.25.123.62:37565)
0928 15:06:06.557364 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.557367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.557617 (+ 250us) server_negotiation.cc:407] Connection header received
0928 15:06:06.558095 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.558096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.558108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.558169 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:06.559181 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.559814 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.561024 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.561276 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.561359 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.563917 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:06.563920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:06.563920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:06.563930 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:06.565510 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.566058 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.566059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.566060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.566125 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.566676 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.566678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.566678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.566819 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:06.566904 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.567211 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.567316 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:06:06.567749 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.557189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:57923)
0928 15:06:06.557459 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.557499 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:06.557982 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:06.558227 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:06.558231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:06.558839 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:06.559056 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.559060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.559945 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.559946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.560915 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.560919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.561405 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.561406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.561530 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.562094 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:06.562104 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:06.563795 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:06.565674 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:06.565686 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:06.565689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:06.565957 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:06.566207 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:06.566207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:06.566208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:06.566571 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:06.566894 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:06.566902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:06.567061 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 15:06:06.567175 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:06:06.567358 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":127,"threads_started":1}
I20260928 15:06:06.568313 9329 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37565
I20260928 15:06:06.568456 9329 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:06.568691 9329 heartbeater.cc:511] Master 127.25.123.62:37565 requested a full tablet report, sending...
I20260928 15:06:06.569721 9137 ts_manager.cc:194] Registered new tserver with Master: 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619)
I20260928 15:06:06.570760 9137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:57923
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:06:06.596024 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37565
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:06.686573 9334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:06.686810 9334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:06.686838 9334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:06.688202 9334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:06.688254 9334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:06.688285 9334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:06:06.690081 9334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:06.690354 9334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:06.690630 9334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:06:06.697404 9334 server_base.cc:1034] running on GCE node
W20260928 15:06:06.697368 9339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:06.697530 9340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:06.697744 9342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:06.698032 9334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:06.698338 9334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:06.699570 9334 hybrid_clock.cc:648] HybridClock initialized: now 1790607966699552 us; error 38 us; skew 500 ppm
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:06.702517 9334 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:06:06.703559 9334 webserver.cc:460] Webserver started at http://127.25.123.2:44823/ using document root <none> and password file <none>
I20260928 15:06:06.704032 9334 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:06.704126 9334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:06.704290 9334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:06.705489 9334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "c35219c4e20d4f648d29ef4da1f3b305"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "8c6bc2cfba4e08ca94499ca39440f401"
server_key_iv: "ca4d958c38a1262e65fbbf3d3d5725b9"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.705879 9334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "c35219c4e20d4f648d29ef4da1f3b305"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "8c6bc2cfba4e08ca94499ca39440f401"
server_key_iv: "ca4d958c38a1262e65fbbf3d3d5725b9"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.707386 9334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:06:06.708532 9349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.708758 9334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:06.708871 9334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "c35219c4e20d4f648d29ef4da1f3b305"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "8c6bc2cfba4e08ca94499ca39440f401"
server_key_iv: "ca4d958c38a1262e65fbbf3d3d5725b9"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.708961 9334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:06.718729 9334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:06.721415 9334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:06.721611 9334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:06.721880 9334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:06.722224 9334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:06.722280 9334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.722317 9334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:06.722358 9334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.729310 9334 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:40573
I20260928 15:06:06.729398 9463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:40573 every 8 connection(s)
I20260928 15:06:06.729776 9334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:06:06.731935 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 9334
I20260928 15:06:06.732033 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:06:06.732355 26092 external_mini_cluster.cc:1428] Setting key a641e8e5906422e0be63b689be6ade2b
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:06:06.744910 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.730644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:52137)
0928 15:06:06.730816 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.730889 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:06.731435 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:06.731705 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:06.731709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:06.732522 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:06.732814 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.732822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.734133 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.734135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.735154 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.735158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.735915 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.735917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.736839 (+ 922us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.737748 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:06.737763 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:06.740079 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:06.742477 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:06.742495 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:06.742499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:06.743087 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:06.743409 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:06.743410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:06.743411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:06.743634 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:06.744035 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:06.744042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:06.744234 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 15:06:06.744341 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:06:06.744656 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260928 15:06:06.744802 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.730706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:52137 (local address 127.25.123.62:37565)
0928 15:06:06.730763 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.730765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.730977 (+ 212us) server_negotiation.cc:407] Connection header received
0928 15:06:06.731553 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.731554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.731566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.731629 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:06.732955 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.733740 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.735470 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.735730 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.735830 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.740308 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:06.740311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:06.740311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:06.740323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:06.742292 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.743214 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.743216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.743217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.743289 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.743724 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.743726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.743726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.743907 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:06.744010 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.744419 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.744524 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 15:06:06.745530 9464 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37565
I20260928 15:06:06.745633 9464 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:06.745810 9464 heartbeater.cc:511] Master 127.25.123.62:37565 requested a full tablet report, sending...
I20260928 15:06:06.746295 9137 ts_manager.cc:194] Registered new tserver with Master: c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573)
I20260928 15:06:06.746730 9137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:52137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:06:06.776291 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37565
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:06.867587 9469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:06.867828 9469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:06.867859 9469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:06.869463 9469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:06.869530 9469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:06.869565 9469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:06:06.871169 9469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:06.871403 9469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:06.871611 9469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:06.877053 9474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:06.877188 9469 server_base.cc:1034] running on GCE node
W20260928 15:06:06.877375 9477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:06.877216 9475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:06.877770 9469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:06.878360 9469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:06.879609 9469 hybrid_clock.cc:648] HybridClock initialized: now 1790607966879606 us; error 128 us; skew 500 ppm
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:06.882622 9469 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:06:06.883169 9469 webserver.cc:460] Webserver started at http://127.25.123.3:46255/ using document root <none> and password file <none>
I20260928 15:06:06.883373 9469 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:06.883414 9469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:06.883510 9469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:06:06.884658 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "3963f1aa44714c59b12676da3aaa8f30"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "4d62a211a77c4413bd6af3ed28c075e9"
server_key_iv: "63cddee9c3b5acd1f496d6239feebd12"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.884972 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "3963f1aa44714c59b12676da3aaa8f30"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "4d62a211a77c4413bd6af3ed28c075e9"
server_key_iv: "63cddee9c3b5acd1f496d6239feebd12"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.886389 9469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:06.887280 9484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.887538 9469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:06.887638 9469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "3963f1aa44714c59b12676da3aaa8f30"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "4d62a211a77c4413bd6af3ed28c075e9"
server_key_iv: "63cddee9c3b5acd1f496d6239feebd12"
server_key_version: "encryptionkey@0"
I20260928 15:06:06.887703 9469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:06.923468 9469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:06.926667 9469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:06.926821 9469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:06.927038 9469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:06.927415 9469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:06.927453 9469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.927477 9469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:06.927524 9469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:06.933914 9469 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:46197
I20260928 15:06:06.933964 9598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:46197 every 8 connection(s)
I20260928 15:06:06.934334 9469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:06:06 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607966, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:06:06.943354 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 9469
I20260928 15:06:06.943483 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:06:06.943666 26092 external_mini_cluster.cc:1428] Setting key 6748883b8d566e399740d9c702ea5fc3
I20260928 15:06:06.945961 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.935240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:44429 (local address 127.25.123.62:37565)
0928 15:06:06.935311 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.935314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.935502 (+ 188us) server_negotiation.cc:407] Connection header received
0928 15:06:06.936004 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.936005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.936015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.936082 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:06.937149 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.937711 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.938809 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.939014 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.939078 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.942127 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:06.942129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:06.942129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:06.942139 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:06.943880 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.944604 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.944605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.944606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.944684 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.945227 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.945228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.945229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.945361 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:06.945430 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.945792 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.945888 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 15:06:06.946198 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.935187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:44429)
0928 15:06:06.935364 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.935409 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:06.935912 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:06.936223 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:06.936227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:06.936838 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:06.937043 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.937048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.937855 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.937857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.938706 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.938708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.939116 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.939117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.939256 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.939967 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:06.939979 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:06.941944 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:06.943986 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:06.943999 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:06.944003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:06.944467 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:06.944919 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:06.944920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:06.944921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:06.945115 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:06.945454 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:06.945462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:06.945618 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 15:06:06.945727 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:06:06.945890 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260928 15:06:06.947022 9599 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37565
I20260928 15:06:06.947117 9599 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:06.947345 9599 heartbeater.cc:511] Master 127.25.123.62:37565 requested a full tablet report, sending...
I20260928 15:06:06.947789 9137 ts_manager.cc:194] Registered new tserver with Master: 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3:46197)
I20260928 15:06:06.948197 9137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:44429
I20260928 15:06:06.956449 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:06:06.965792 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.958185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.25.123.62:37565)
0928 15:06:06.958368 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.958371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.958463 (+ 92us) server_negotiation.cc:407] Connection header received
0928 15:06:06.958569 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.958571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.958582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.958651 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:06.959292 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.959811 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.960637 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.960836 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.960927 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.962238 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:06.962241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:06.962241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:06.962249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:06.963816 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.964546 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.964547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.964547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.964733 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:06.965150 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:06.965152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:06.965152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:06.965330 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:06.965519 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.965560 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.965672 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260928 15:06:06.968292 9137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:06:06.970134 9137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:06:06.979532 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.973834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40573 (local address 127.0.0.1:46568)
0928 15:06:06.974051 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.974096 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:06.974290 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:06.975215 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:06.975216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:06.975219 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:06.975328 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.975330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.976318 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.976320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.977710 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.977717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.978740 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.978742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.978875 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.978895 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 15:06:06.979211 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 15:06:06.979348 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260928 15:06:06.979887 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.974048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.25.123.2:40573)
0928 15:06:06.974981 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.974985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.974995 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:06:06.975035 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.975037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.975053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.975127 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:06.975413 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.976184 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.977914 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.978611 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.978731 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.978826 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.979254 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.979818 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":105,"threads_started":1}
I20260928 15:06:06.980384 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.974221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:46197 (local address 127.0.0.1:36850)
0928 15:06:06.974345 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.974355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:06.974399 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:06.974914 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:06.974915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:06.974921 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:06.975019 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.975022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.977225 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.977225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.978709 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.978713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.980103 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.980105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.980203 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.980211 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:06:06.980250 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 15:06:06.980309 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":12,"thread_start_us":89,"threads_started":1}
I20260928 15:06:06.980573 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.973834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:46619 (local address 127.0.0.1:44312)
0928 15:06:06.974166 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:06.974177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:06.974256 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:06.974622 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:06.974625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:06.974628 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:06.974721 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.974723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.977221 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.977222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.978708 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:06.978713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.980302 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.980303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:06.980372 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.980377 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:06:06.980455 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:06:06.980503 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":147,"spinlock_wait_cycles":768,"thread_start_us":98,"threads_started":1}
I20260928 15:06:06.980700 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.973964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.25.123.1:46619)
0928 15:06:06.974265 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.974271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.974284 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:06:06.974367 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.974371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.974388 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.974444 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:06.974910 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.977054 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.979093 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.980228 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.980297 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.980498 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.980534 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.980588 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260928 15:06:06.980911 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:06.974446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.25.123.3:46197)
0928 15:06:06.974669 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:06.974673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:06.974688 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:06:06.974727 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:06.974729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:06.974751 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:06.974839 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:06.975186 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.977089 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.978962 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:06.979973 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:06.980608 (+ 635us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:06.980748 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:06.980780 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 15:06:06.980845 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":112,"threads_started":1}
I20260928 15:06:06.980801 9397 tablet_service.cc:1467] Processing CreateTablet for tablet 46c4445b7c5a4a5895d71746551a4366 (DEFAULT_TABLE table=test-table [id=794b1dd6eb26475ca2f33b5ff5f9c9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:06.981240 9397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c4445b7c5a4a5895d71746551a4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:06.981609 9532 tablet_service.cc:1467] Processing CreateTablet for tablet 46c4445b7c5a4a5895d71746551a4366 (DEFAULT_TABLE table=test-table [id=794b1dd6eb26475ca2f33b5ff5f9c9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:06.982054 9532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c4445b7c5a4a5895d71746551a4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:06.985486 9616 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Bootstrap starting.
I20260928 15:06:06.985383 9262 tablet_service.cc:1467] Processing CreateTablet for tablet 46c4445b7c5a4a5895d71746551a4366 (DEFAULT_TABLE table=test-table [id=794b1dd6eb26475ca2f33b5ff5f9c9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:06.986095 9262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46c4445b7c5a4a5895d71746551a4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:06.986817 9617 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Bootstrap starting.
I20260928 15:06:06.986899 9616 tablet_bootstrap.cc:654] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:06.987534 9617 tablet_bootstrap.cc:654] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:06.987622 9616 log.cc:824] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:06.988229 9617 log.cc:824] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:06.988754 9616 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: No bootstrap required, opened a new log
I20260928 15:06:06.988852 9616 ts_tablet_manager.cc:1397] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 15:06:06.989244 9617 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: No bootstrap required, opened a new log
I20260928 15:06:06.989344 9617 ts_tablet_manager.cc:1397] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:06:06.991694 9620 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap starting.
I20260928 15:06:06.991719 9616 raft_consensus.cc:348] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:06.991987 9617 raft_consensus.cc:348] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:06.992151 9616 raft_consensus.cc:374] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:06.992219 9616 raft_consensus.cc:729] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35219c4e20d4f648d29ef4da1f3b305, State: Initialized, Role: FOLLOWER
I20260928 15:06:06.992224 9617 raft_consensus.cc:374] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:06.992262 9617 raft_consensus.cc:729] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963f1aa44714c59b12676da3aaa8f30, State: Initialized, Role: FOLLOWER
I20260928 15:06:06.992460 9617 consensus_queue.cc:260] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:06.992470 9616 consensus_queue.cc:260] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:06.992871 9616 ts_tablet_manager.cc:1428] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:06:06.992883 9599 heartbeater.cc:503] Master 127.25.123.62:37565 was elected leader, sending a full tablet report...
I20260928 15:06:06.993002 9620 tablet_bootstrap.cc:654] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:06.993214 9464 heartbeater.cc:503] Master 127.25.123.62:37565 was elected leader, sending a full tablet report...
I20260928 15:06:06.992872 9617 ts_tablet_manager.cc:1428] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 15:06:06.993829 9620 log.cc:824] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:06.994807 9620 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: No bootstrap required, opened a new log
I20260928 15:06:06.994904 9620 ts_tablet_manager.cc:1397] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:06.996899 9620 raft_consensus.cc:348] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:06.997270 9620 raft_consensus.cc:374] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:06.997337 9620 raft_consensus.cc:729] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d3bc139e744aa8867d2956b2ec418c, State: Initialized, Role: FOLLOWER
I20260928 15:06:06.997499 9620 consensus_queue.cc:260] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:06.997869 9620 ts_tablet_manager.cc:1428] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:06:06.998034 9329 heartbeater.cc:503] Master 127.25.123.62:37565 was elected leader, sending a full tablet report...
I20260928 15:06:07.022095 9622 raft_consensus.cc:482] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:06:07.022244 9622 raft_consensus.cc:504] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:07.022704 9622 leader_election.cc:288] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619), 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3:46197)
I20260928 15:06:07.027236 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.023743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:36249 (local address 127.25.123.3:46197)
0928 15:06:07.023968 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:07.023970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:07.023980 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:06:07.024010 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:07.024011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:07.024020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:07.024164 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:07.024694 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.025296 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.026400 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.026814 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.026921 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:07.026933 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:07.027100 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 15:06:07.027146 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 15:06:07.027240 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.023231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:46197 (local address 127.25.123.2:36249)
0928 15:06:07.023526 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:07.023537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:07.023625 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:07.024181 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:07.024182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:07.024185 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:07.024371 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:07.024380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.025386 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.025388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:07.026291 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:07.026294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.026921 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.026922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:07.027008 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:07.027012 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:06:07.027091 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:06:07.027140 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":20,"thread_start_us":182,"threads_started":1}
I20260928 15:06:07.027352 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.023101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:46619 (local address 127.25.123.2:57175)
0928 15:06:07.023475 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:07.023491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:07.023579 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:07.023733 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:07.023735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:07.023737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:07.023890 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:07.023893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.025135 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.025137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:07.026296 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:07.026299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.027059 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.027060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:07.027152 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:07.027156 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:06:07.027231 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:06:07.027274 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":3}
I20260928 15:06:07.027354 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.023379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:57175 (local address 127.25.123.1:46619)
0928 15:06:07.023499 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:07.023502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:07.023586 (+ 84us) server_negotiation.cc:407] Connection header received
0928 15:06:07.023651 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:07.023653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:07.023665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:07.023730 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:07.023986 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.024953 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.026458 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.026954 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.027059 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:07.027069 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:07.027238 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 15:06:07.027285 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:06:07.027714 9552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c4445b7c5a4a5895d71746551a4366" candidate_uuid: "c35219c4e20d4f648d29ef4da1f3b305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963f1aa44714c59b12676da3aaa8f30" is_pre_election: true
I20260928 15:06:07.027817 9283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c4445b7c5a4a5895d71746551a4366" candidate_uuid: "c35219c4e20d4f648d29ef4da1f3b305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d3bc139e744aa8867d2956b2ec418c" is_pre_election: true
I20260928 15:06:07.027901 9552 raft_consensus.cc:2454] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35219c4e20d4f648d29ef4da1f3b305 in term 0.
I20260928 15:06:07.027987 9283 raft_consensus.cc:2454] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35219c4e20d4f648d29ef4da1f3b305 in term 0.
I20260928 15:06:07.028156 9351 leader_election.cc:302] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963f1aa44714c59b12676da3aaa8f30, c35219c4e20d4f648d29ef4da1f3b305; no voters:
I20260928 15:06:07.028378 9622 raft_consensus.cc:2781] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:06:07.028481 9622 raft_consensus.cc:482] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:06:07.028528 9622 raft_consensus.cc:3037] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:07.029296 9622 raft_consensus.cc:504] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:07.029422 9622 leader_election.cc:288] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [CANDIDATE]: Term 1 election: Requested vote from peers 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619), 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3:46197)
I20260928 15:06:07.029650 9283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c4445b7c5a4a5895d71746551a4366" candidate_uuid: "c35219c4e20d4f648d29ef4da1f3b305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d3bc139e744aa8867d2956b2ec418c"
I20260928 15:06:07.029738 9283 raft_consensus.cc:3037] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:07.029709 9552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46c4445b7c5a4a5895d71746551a4366" candidate_uuid: "c35219c4e20d4f648d29ef4da1f3b305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963f1aa44714c59b12676da3aaa8f30"
I20260928 15:06:07.029817 9552 raft_consensus.cc:3037] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:07.030493 9283 raft_consensus.cc:2454] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35219c4e20d4f648d29ef4da1f3b305 in term 1.
I20260928 15:06:07.030576 9552 raft_consensus.cc:2454] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35219c4e20d4f648d29ef4da1f3b305 in term 1.
I20260928 15:06:07.030705 9351 leader_election.cc:302] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d3bc139e744aa8867d2956b2ec418c, c35219c4e20d4f648d29ef4da1f3b305; no voters:
I20260928 15:06:07.030792 9622 raft_consensus.cc:2781] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:06:07.030982 9622 raft_consensus.cc:686] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 1 LEADER]: Becoming Leader. State: Replica: c35219c4e20d4f648d29ef4da1f3b305, State: Running, Role: LEADER
I20260928 15:06:07.031071 9622 consensus_queue.cc:237] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:07.032105 9136 catalog_manager.cc:5495] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 reported cstate change: term changed from 0 to 1, leader changed from <none> to c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2). New cstate: current_term: 1 leader_uuid: "c35219c4e20d4f648d29ef4da1f3b305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } health_report { overall_health: UNKNOWN } } }
I20260928 15:06:07.041574 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:07.037898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.25.123.2:40573)
0928 15:06:07.037988 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:07.037990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:07.038008 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:06:07.038063 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:07.038064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:07.038086 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:07.038164 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:06:07.038340 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.038882 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.040062 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:07.040320 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:07.040458 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:07.040803 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:06:07.040978 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:06:07.041118 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:07.041384 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:06:07.041418 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:06:07.044456 9552 raft_consensus.cc:1261] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Refusing update from remote peer c35219c4e20d4f648d29ef4da1f3b305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:06:07.044456 9283 raft_consensus.cc:1261] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Refusing update from remote peer c35219c4e20d4f648d29ef4da1f3b305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:06:07.044878 9626 consensus_queue.cc:1035] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:06:07.045032 9622 consensus_queue.cc:1035] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:06:07.047719 9630 mvcc.cc:205] Tried to move back new op lower bound from 7334330233010528256 to 7334330232960024576. Current Snapshot: MvccSnapshot[applied={T|T < 7334330233010528256}]
I20260928 15:06:07.050925 9137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:07.051074 9137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:07.052148 9137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Sending DeleteTablet for 3 replicas of tablet 46c4445b7c5a4a5895d71746551a4366
I20260928 15:06:07.052681 9397 tablet_service.cc:1514] Processing DeleteTablet for tablet 46c4445b7c5a4a5895d71746551a4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:07 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:46568
I20260928 15:06:07.052940 9262 tablet_service.cc:1514] Processing DeleteTablet for tablet 46c4445b7c5a4a5895d71746551a4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:07 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:44312
I20260928 15:06:07.052940 9532 tablet_service.cc:1514] Processing DeleteTablet for tablet 46c4445b7c5a4a5895d71746551a4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:07 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36850
I20260928 15:06:07.053112 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 9199
I20260928 15:06:07.056200 9639 tablet_replica.cc:330] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: stopping tablet replica
I20260928 15:06:07.056430 9639 raft_consensus.cc:2229] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:06:07.056926 9639 raft_consensus.cc:2258] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:07.057281 9641 tablet_replica.cc:330] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: stopping tablet replica
I20260928 15:06:07.057430 9641 raft_consensus.cc:2229] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:07.057554 9641 raft_consensus.cc:2258] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:07.057767 9639 ts_tablet_manager.cc:1905] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:07.060747 9639 ts_tablet_manager.cc:1918] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:07.060878 9639 log.cc:1196] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/46c4445b7c5a4a5895d71746551a4366
I20260928 15:06:07.061443 9639 ts_tablet_manager.cc:1939] T 46c4445b7c5a4a5895d71746551a4366 P c35219c4e20d4f648d29ef4da1f3b305: Deleting consensus metadata
I20260928 15:06:07.061753 9641 ts_tablet_manager.cc:1905] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:07.062204 9122 catalog_manager.cc:4841] TS c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573): tablet 46c4445b7c5a4a5895d71746551a4366 (table test-table [id=794b1dd6eb26475ca2f33b5ff5f9c9a8]) successfully deleted
W20260928 15:06:07.063356 9121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:46619 (error 108)
W20260928 15:06:07.064106 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:06:07.063838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:46619 (local address 127.0.0.1:44320)
0928 15:06:07.063907 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:07.063919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:07.063974 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:07.064045 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:46619: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 15:06:07.064250 9121 catalog_manager.cc:4584] TS 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46c4445b7c5a4a5895d71746551a4366: Network error: Client connection negotiation failed: client connection to 127.25.123.1:46619: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 15:06:07.064417 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 9334
I20260928 15:06:07.064453 9641 ts_tablet_manager.cc:1918] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:07.066282 9641 log.cc:1196] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/46c4445b7c5a4a5895d71746551a4366
I20260928 15:06:07.066639 9641 ts_tablet_manager.cc:1939] T 46c4445b7c5a4a5895d71746551a4366 P 3963f1aa44714c59b12676da3aaa8f30: Deleting consensus metadata
I20260928 15:06:07.067188 9121 catalog_manager.cc:4841] TS 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3:46197): tablet 46c4445b7c5a4a5895d71746551a4366 (table test-table [id=794b1dd6eb26475ca2f33b5ff5f9c9a8]) successfully deleted
I20260928 15:06:07.070178 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 9469
I20260928 15:06:07.075042 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 9104
I20260928 15:06:27.082576 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37565
--webserver_interface=127.25.123.62
--webserver_port=36787
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:37565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:27.172180 9643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:27.172503 9643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:27.172557 9643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:27.173905 9643 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:06:27.173986 9643 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:06:27.174013 9643 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:06:27.174042 9643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:06:27.174068 9643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:27.174098 9643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:27.174125 9643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:06:27.174149 9643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:06:27.175966 9643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:37565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=36787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:27.176294 9643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:27.176582 9643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:27.185909 9650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:27.186062 9643 server_base.cc:1034] running on GCE node
W20260928 15:06:27.185871 9652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:27.185871 9649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:27.186527 9643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:27.186885 9643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:27.188076 9643 hybrid_clock.cc:648] HybridClock initialized: now 1790607987188045 us; error 48 us; skew 500 ppm
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:27.202284 9643 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:06:27.202880 9643 webserver.cc:460] Webserver started at http://127.25.123.62:36787/ using document root <none> and password file <none>
I20260928 15:06:27.203148 9643 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:27.203244 9643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:27.204847 9643 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:06:27.205777 9659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.206061 9643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:06:27.206179 9643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "a5c510f895966c3c6528d3b805d0ee63"
server_key_iv: "19e83b5082378ca0c178d27b8b91c75d"
server_key_version: "encryptionkey@0"
I20260928 15:06:27.206614 9643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:27.220598 9643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:27.223095 9643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:27.223222 9643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:27.227994 9643 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:37565
I20260928 15:06:27.228034 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:37565 every 8 connection(s)
I20260928 15:06:27.228976 9643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:06:27.230702 9712 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:06:27.231521 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Bootstrap starting.
I20260928 15:06:27.238914 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 9643
I20260928 15:06:27.239296 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:46619
--local_ip_for_outbound_sockets=127.25.123.1
--tserver_master_addrs=127.25.123.62:37565
--webserver_port=43961
--webserver_interface=127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:27.241201 9712 log.cc:824] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:27.243278 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:27.243615 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Bootstrap complete.
I20260928 15:06:27.246094 9712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:27.246367 9712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea32c79a3d84c7991bcb2a7193cfcc5, State: Initialized, Role: FOLLOWER
I20260928 15:06:27.246585 9712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:27.246690 9712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:06:27.246754 9712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:06:27.246804 9712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:06:27.247913 9712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:27.248049 9712 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ea32c79a3d84c7991bcb2a7193cfcc5; no voters:
I20260928 15:06:27.248301 9712 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:06:27.248346 9716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:06:27.248492 9716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 LEADER]: Becoming Leader. State: Replica: 9ea32c79a3d84c7991bcb2a7193cfcc5, State: Running, Role: LEADER
I20260928 15:06:27.248649 9716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:27.248654 9712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:06:27.249763 9717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } } }
I20260928 15:06:27.249788 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea32c79a3d84c7991bcb2a7193cfcc5. Latest consensus state: current_term: 2 leader_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } } }
I20260928 15:06:27.249864 9717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:27.249890 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:27.250066 9728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:06:27.251365 9728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:06:27.251506 9728 catalog_manager.cc:1259] Loaded cluster ID: 445b375ede744372ac883e5657e03b5e
I20260928 15:06:27.251540 9728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:06:27.252878 9728 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:06:27.253075 9728 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Loaded TSK: 0
I20260928 15:06:27.266185 9728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Generated new TSK 1
I20260928 15:06:27.266322 9728 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:06:27.345098 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:27.345453 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:27.345511 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:27.347788 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:27.347865 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:27.347919 9714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:06:27.350337 9714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:46619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=43961
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:27.350658 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:27.350976 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:06:27.358548 9714 server_base.cc:1034] running on GCE node
W20260928 15:06:27.358325 9739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:27.361280 9740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:27.364567 9742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:27.364996 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:27.365404 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:27.366578 9714 hybrid_clock.cc:648] HybridClock initialized: now 1790607987366555 us; error 39 us; skew 500 ppm
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:27.370339 9714 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:06:27.370976 9714 webserver.cc:460] Webserver started at http://127.25.123.1:43961/ using document root <none> and password file <none>
I20260928 15:06:27.371310 9714 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:27.371402 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:27.373502 9714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:27.374420 9750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.374593 9714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:27.374693 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "30d3bc139e744aa8867d2956b2ec418c"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "0166ebd19b8ee35fcc3b051ff770a801"
server_key_iv: "f38f0e70c53d4c43e4087d9697711ec6"
server_key_version: "encryptionkey@0"
I20260928 15:06:27.375347 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:27.443719 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:27.446226 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:27.446352 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:27.446579 9714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:27.447059 9757 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:06:27.448670 9714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:06:27.448726 9714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:06:27.448755 9714 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:06:27.449582 9714 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:06:27.449618 9714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:06:27.449790 9757 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap starting.
I20260928 15:06:27.457156 9714 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:46619
I20260928 15:06:27.457283 9864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:46619 every 8 connection(s)
I20260928 15:06:27.457600 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:06:27.459697 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 9714
I20260928 15:06:27.460072 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:40573
--local_ip_for_outbound_sockets=127.25.123.2
--tserver_master_addrs=127.25.123.62:37565
--webserver_port=44823
--webserver_interface=127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:27.462136 9757 log.cc:824] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:27.465015 9757 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:27.465341 9757 tablet_bootstrap.cc:492] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap complete.
I20260928 15:06:27.466548 9757 ts_tablet_manager.cc:1397] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:06:27.467629 9757 raft_consensus.cc:348] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:27.467947 9757 raft_consensus.cc:729] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d3bc139e744aa8867d2956b2ec418c, State: Initialized, Role: FOLLOWER
I20260928 15:06:27.468109 9757 consensus_queue.cc:260] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } }
I20260928 15:06:27.468479 9757 ts_tablet_manager.cc:1428] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 15:06:27.472002 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.458471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:43545 (local address 127.25.123.62:37565)
0928 15:06:27.458630 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.458639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.458682 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:06:27.459734 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.459743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.459827 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.460000 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:27.461092 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.462289 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.463838 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.464487 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.465072 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.467812 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:27.467817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:27.467817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:27.467831 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:27.469855 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.470433 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.470436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.470436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.470576 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.470912 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.470914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.470914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.471218 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:27.471305 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.471721 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.471849 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260928 15:06:27.472008 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.458298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:43545)
0928 15:06:27.458517 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.458577 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.459372 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.460073 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.460079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.460613 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:27.460920 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.460929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.462465 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.462466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.463727 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.463730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.464626 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.464627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.464770 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.465650 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:27.465661 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:27.467673 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:27.469958 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.469966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.469968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.470331 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.470657 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.470658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.470658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.470810 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.471309 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:27.471317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:27.471500 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.471622 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.471801 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
I20260928 15:06:27.472956 9865 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37565
I20260928 15:06:27.473067 9865 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:27.473274 9865 heartbeater.cc:511] Master 127.25.123.62:37565 requested a full tablet report, sending...
I20260928 15:06:27.474107 9676 ts_manager.cc:194] Registered new tserver with Master: 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619)
I20260928 15:06:27.475296 9676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:43545
I20260928 15:06:27.476549 9865 heartbeater.cc:503] Master 127.25.123.62:37565 was elected leader, sending a full tablet report...
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:06:27.484474 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.474778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:46619 (local address 127.0.0.1:55456)
0928 15:06:27.474966 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.475006 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.475081 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.476019 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.476026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.476365 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:27.476476 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.476479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.477621 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.477623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.478272 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.478274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.478799 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.478800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.478890 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.479457 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:27.479463 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:27.481130 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:27.483054 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.483062 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.483064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.483354 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.483637 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.483637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.483638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.483706 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.484033 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:27.484047 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:27.484127 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.484212 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.484341 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260928 15:06:27.484499 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.475669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.25.123.1:46619)
0928 15:06:27.475861 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.475866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.475876 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:06:27.475912 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.475915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.475933 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.476017 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:27.476593 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.477518 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.478375 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.478709 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.478800 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.481243 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:27.481250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:27.481251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:27.481258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:27.482950 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.483472 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.483476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.483477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.483555 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.483773 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.483774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.483774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.483961 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:27.484040 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.484244 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.484332 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":157,"thread_start_us":125,"threads_started":1}
I20260928 15:06:27.484970 9796 tablet_service.cc:1514] Processing DeleteTablet for tablet 46c4445b7c5a4a5895d71746551a4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:07 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:55456
I20260928 15:06:27.485570 9874 tablet_replica.cc:330] stopping tablet replica
I20260928 15:06:27.485704 9874 raft_consensus.cc:2229] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:27.485809 9874 raft_consensus.cc:2258] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:27.486148 9874 ts_tablet_manager.cc:1905] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:27.488505 9874 ts_tablet_manager.cc:1918] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:27.488605 9874 log.cc:1196] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal/wals/46c4445b7c5a4a5895d71746551a4366
I20260928 15:06:27.488843 9874 ts_tablet_manager.cc:1939] T 46c4445b7c5a4a5895d71746551a4366 P 30d3bc139e744aa8867d2956b2ec418c: Deleting consensus metadata
I20260928 15:06:27.489464 9661 catalog_manager.cc:4841] TS 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619): tablet 46c4445b7c5a4a5895d71746551a4366 (table test-table [id=794b1dd6eb26475ca2f33b5ff5f9c9a8]) successfully deleted
W20260928 15:06:27.552757 9869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:27.553023 9869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:27.553092 9869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:27.554379 9869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:27.554466 9869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:27.554512 9869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:06:27.556114 9869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:40573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=44823
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:27.556437 9869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:27.556699 9869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:27.562316 9880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:27.562328 9881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:27.562316 9883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:27.562619 9869 server_base.cc:1034] running on GCE node
I20260928 15:06:27.562922 9869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:27.563237 9869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:27.564493 9869 hybrid_clock.cc:648] HybridClock initialized: now 1790607987564464 us; error 43 us; skew 500 ppm
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:27.567497 9869 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:06:27.568041 9869 webserver.cc:460] Webserver started at http://127.25.123.2:44823/ using document root <none> and password file <none>
I20260928 15:06:27.568270 9869 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:27.568329 9869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:27.570014 9869 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:06:27.570814 9890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.571122 9869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:06:27.571249 9869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "c35219c4e20d4f648d29ef4da1f3b305"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "8c6bc2cfba4e08ca94499ca39440f401"
server_key_iv: "ca4d958c38a1262e65fbbf3d3d5725b9"
server_key_version: "encryptionkey@0"
I20260928 15:06:27.571623 9869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:27.594923 9869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:27.597105 9869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:27.597234 9869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:27.597474 9869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:27.597774 9869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:27.597806 9869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.597828 9869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:27.597841 9869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.604023 9869 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:40573
I20260928 15:06:27.604202 10004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:40573 every 8 connection(s)
I20260928 15:06:27.604542 9869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:06:27.606516 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 9869
I20260928 15:06:27.606822 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:46197
--local_ip_for_outbound_sockets=127.25.123.3
--tserver_master_addrs=127.25.123.62:37565
--webserver_port=46255
--webserver_interface=127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:06:27.615855 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.605342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:51039)
0928 15:06:27.605499 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.605536 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.606062 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.606355 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.606359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.607092 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:27.607302 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.607307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.608128 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.608128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.609106 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.609110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.609701 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.609702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.609828 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.610522 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:27.610533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:27.612121 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:27.614168 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.614175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.614177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.614407 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.614694 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.614695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.614695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.614825 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.615177 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:27.615183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:27.615411 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.615518 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.615672 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260928 15:06:27.615859 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.605343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:51039 (local address 127.25.123.62:37565)
0928 15:06:27.605416 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.605418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.605635 (+ 217us) server_negotiation.cc:407] Connection header received
0928 15:06:27.606173 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.606175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.606191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.606255 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:27.607410 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.608013 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.609220 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.609584 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.609667 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.612235 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:27.612237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:27.612238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:27.612247 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:27.614056 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.614487 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.614489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.614489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.614558 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.614905 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.614906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.614907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.615086 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:27.615168 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.615597 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.615714 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:06:27.616829 10005 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37565
I20260928 15:06:27.616952 10005 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:27.617175 10005 heartbeater.cc:511] Master 127.25.123.62:37565 requested a full tablet report, sending...
I20260928 15:06:27.617735 9676 ts_manager.cc:194] Registered new tserver with Master: c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573)
I20260928 15:06:27.618146 9676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:51039
W20260928 15:06:27.694155 10008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:27.694460 10008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:27.694509 10008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:27.696461 10008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:27.696566 10008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:27.696619 10008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:06:27.699416 10008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:46197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=46255
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:27.699800 10008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:27.700079 10008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:27.705775 10013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:27.705996 10014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:27.706137 10008 server_base.cc:1034] running on GCE node
W20260928 15:06:27.705773 10016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:27.706568 10008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:27.706911 10008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:27.708128 10008 hybrid_clock.cc:648] HybridClock initialized: now 1790607987708096 us; error 47 us; skew 500 ppm
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:27.711302 10008 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:06:27.711807 10008 webserver.cc:460] Webserver started at http://127.25.123.3:46255/ using document root <none> and password file <none>
I20260928 15:06:27.712090 10008 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:27.712179 10008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:27.713874 10008 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:27.714689 10023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.714933 10008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:27.715076 10008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "3963f1aa44714c59b12676da3aaa8f30"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "4d62a211a77c4413bd6af3ed28c075e9"
server_key_iv: "63cddee9c3b5acd1f496d6239feebd12"
server_key_version: "encryptionkey@0"
I20260928 15:06:27.715472 10008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:27.734985 10008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:27.737660 10008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:27.737849 10008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:27.738175 10008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:27.738519 10008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:27.738576 10008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.738613 10008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:27.738651 10008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:27.744956 10008 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:46197
I20260928 15:06:27.744992 10137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:46197 every 8 connection(s)
I20260928 15:06:27.745371 10008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:06:27 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607987, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:06:27.752765 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10008
I20260928 15:06:27.756490 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.746176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:37241 (local address 127.25.123.62:37565)
0928 15:06:27.746249 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.746251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.746501 (+ 250us) server_negotiation.cc:407] Connection header received
0928 15:06:27.746977 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.746978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.746989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.747053 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:27.748019 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.748647 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.750047 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.750287 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.750353 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.752881 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:27.752884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:27.752884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:27.752894 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:27.754735 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.755184 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.755185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.755185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.755251 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.755654 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.755655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.755655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.755770 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:27.755834 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.756203 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.756325 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:06:27.756662 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.746172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:37241)
0928 15:06:27.746340 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.746385 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.746888 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.747105 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.747108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.747696 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:27.747921 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.747925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.748762 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.748764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.749920 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.749923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.750385 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.750386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.750559 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.751146 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:27.751155 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:27.752761 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:27.754838 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.754845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.754847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.755106 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.755338 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:27.755339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:27.755340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:27.755483 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:27.755849 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:27.755856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:27.756021 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.756160 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.756341 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 15:06:27.757227 10138 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37565
I20260928 15:06:27.757347 10138 heartbeater.cc:464] Registering TS with master...
I20260928 15:06:27.757555 10138 heartbeater.cc:511] Master 127.25.123.62:37565 requested a full tablet report, sending...
I20260928 15:06:27.758086 9676 ts_manager.cc:194] Registered new tserver with Master: 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3:46197)
I20260928 15:06:27.758597 9676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:37241
I20260928 15:06:27.761099 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.753369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.25.123.62:37565)
0928 15:06:27.753535 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.753537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.753682 (+ 145us) server_negotiation.cc:407] Connection header received
0928 15:06:27.753713 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.753714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.753727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.753808 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:27.754577 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.755244 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.756263 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.756603 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.756675 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.758386 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:27.758388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:27.758388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:27.758394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:27.759846 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.760208 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.760210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.760210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.760272 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.760509 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.760510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.760511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.760625 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:27.760833 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.760886 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.760981 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260928 15:06:27.762055 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:06:27.765447 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.762752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.25.123.62:37565)
0928 15:06:27.762837 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.762839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.762926 (+ 87us) server_negotiation.cc:407] Connection header received
0928 15:06:27.763008 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.763009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.763054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.763144 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:06:27.763295 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.763793 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.764624 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.764868 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.764952 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.765124 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:06:27.765179 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 15:06:27.765370 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46128: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 15:06:27.765523 10142 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46128: token expired
W20260928 15:06:27.765642 9604 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.25.123.62:37565)
I20260928 15:06:27.772550 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.765901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.25.123.62:37565)
0928 15:06:27.765981 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.765983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.766008 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:06:27.766026 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.766027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.766035 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.766090 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:27.766806 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.767306 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.768044 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.768237 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.768321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.769551 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:27.769552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:27.769553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:27.769561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:27.771096 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.771571 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.771573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.771573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.771642 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:27.771870 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:27.771871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:27.771872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:27.772006 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:27.772138 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.772248 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.772338 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:06:27.774099 9676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:06:27.775903 9676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:06:27.779047 9796 tablet_service.cc:1467] Processing CreateTablet for tablet 4410fa7a351349dda608ca35e727a39b (DEFAULT_TABLE table=test-table [id=124718cbc9cd45c19663ff64c4ba02f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:27.779274 9796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4410fa7a351349dda608ca35e727a39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:27.781013 9757 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap starting.
I20260928 15:06:27.781383 9757 tablet_bootstrap.cc:654] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:27.782337 9757 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: No bootstrap required, opened a new log
I20260928 15:06:27.782397 9757 ts_tablet_manager.cc:1397] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:27.782557 9757 raft_consensus.cc:348] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.782642 9757 raft_consensus.cc:374] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:27.782692 9757 raft_consensus.cc:729] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d3bc139e744aa8867d2956b2ec418c, State: Initialized, Role: FOLLOWER
I20260928 15:06:27.782749 9757 consensus_queue.cc:260] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.782847 9757 ts_tablet_manager.cc:1428] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 15:06:27.782886 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.778483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40573 (local address 127.0.0.1:36866)
0928 15:06:27.778606 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.778619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.778674 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.779103 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.779104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.779106 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:27.779142 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.779144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.780487 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.780488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.781619 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.781629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.782582 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.782583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.782653 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.782681 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.782746 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.782800 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":88,"threads_started":1}
I20260928 15:06:27.782958 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.778439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:46197 (local address 127.0.0.1:49378)
0928 15:06:27.778498 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.778511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.778558 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.779030 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.779031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.779033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:27.779099 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.779101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.780326 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.780327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.781795 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.781799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.782703 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.782705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.782782 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.782786 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.782842 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.782895 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 15:06:27.783195 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.778660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.25.123.3:46197)
0928 15:06:27.778833 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.778838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.778849 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:06:27.778894 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.778895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.778914 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.779023 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:27.779169 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.780218 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.781899 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.782599 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.782697 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.782748 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.783071 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.783118 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260928 15:06:27.783533 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.778731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.25.123.2:40573)
0928 15:06:27.778907 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.778910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.778919 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:06:27.778951 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.778953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.778966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.779261 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:27.779344 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.780354 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.781713 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.782486 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.782579 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.782648 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.783116 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.783161 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 15:06:27.784010 10071 tablet_service.cc:1467] Processing CreateTablet for tablet 4410fa7a351349dda608ca35e727a39b (DEFAULT_TABLE table=test-table [id=124718cbc9cd45c19663ff64c4ba02f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:27.784157 9939 tablet_service.cc:1467] Processing CreateTablet for tablet 4410fa7a351349dda608ca35e727a39b (DEFAULT_TABLE table=test-table [id=124718cbc9cd45c19663ff64c4ba02f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:06:27.784472 9939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4410fa7a351349dda608ca35e727a39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:27.784592 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4410fa7a351349dda608ca35e727a39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:27.788551 10151 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Bootstrap starting.
I20260928 15:06:27.789315 10151 tablet_bootstrap.cc:654] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:27.789848 10151 log.cc:824] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:27.790208 10152 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Bootstrap starting.
I20260928 15:06:27.790771 10151 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: No bootstrap required, opened a new log
I20260928 15:06:27.790856 10151 ts_tablet_manager.cc:1397] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:06:27.790894 10152 tablet_bootstrap.cc:654] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Neither blocks nor log segments found. Creating new log.
I20260928 15:06:27.791503 10152 log.cc:824] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:27.792387 10152 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: No bootstrap required, opened a new log
I20260928 15:06:27.792495 10152 ts_tablet_manager.cc:1397] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:06:27.793072 10151 raft_consensus.cc:348] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.793273 10151 raft_consensus.cc:374] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:27.793322 10151 raft_consensus.cc:729] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963f1aa44714c59b12676da3aaa8f30, State: Initialized, Role: FOLLOWER
I20260928 15:06:27.793452 10151 consensus_queue.cc:260] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.793787 10151 ts_tablet_manager.cc:1428] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260928 15:06:27.793803 10138 heartbeater.cc:503] Master 127.25.123.62:37565 was elected leader, sending a full tablet report...
I20260928 15:06:27.794410 10152 raft_consensus.cc:348] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.794603 10152 raft_consensus.cc:374] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:06:27.794639 10152 raft_consensus.cc:729] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35219c4e20d4f648d29ef4da1f3b305, State: Initialized, Role: FOLLOWER
I20260928 15:06:27.794755 10152 consensus_queue.cc:260] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.795037 10152 ts_tablet_manager.cc:1428] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:06:27.795054 10005 heartbeater.cc:503] Master 127.25.123.62:37565 was elected leader, sending a full tablet report...
I20260928 15:06:27.806881 10155 raft_consensus.cc:482] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:06:27.807029 10155 raft_consensus.cc:504] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.807462 10155 leader_election.cc:288] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573), 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619)
I20260928 15:06:27.811101 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.807731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:46663 (local address 127.25.123.2:40573)
0928 15:06:27.807780 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.807782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.807794 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:06:27.807861 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.807862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.807873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.807932 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:27.808128 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.808979 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.810152 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.810717 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.810810 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.810820 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.810980 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.811021 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:06:27.811146 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.807660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40573 (local address 127.25.123.3:46663)
0928 15:06:27.807735 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.807747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.807795 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.807952 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.807952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.807954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:27.808039 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.808041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.809094 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.809095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.810029 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.810032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.810807 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.810808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.810901 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.810904 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.810971 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.811018 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:06:27.811164 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.807781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:39199 (local address 127.25.123.1:46619)
0928 15:06:27.807827 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.807829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.808001 (+ 172us) server_negotiation.cc:407] Connection header received
0928 15:06:27.808069 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.808069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.808111 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.808182 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:06:27.808317 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.809070 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.810330 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.810796 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.810901 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.810988 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.811036 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.811074 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:06:27.811336 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.807728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:46619 (local address 127.25.123.3:39199)
0928 15:06:27.807895 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.807904 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.807960 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.808198 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:27.808198 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:27.808200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:06:27.808244 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.808245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.809135 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.809136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.810195 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:27.810199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.810905 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.810906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:27.810975 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.810978 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:06:27.811031 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:06:27.811072 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":12,"thread_start_us":116,"threads_started":1}
I20260928 15:06:27.811542 9959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4410fa7a351349dda608ca35e727a39b" candidate_uuid: "3963f1aa44714c59b12676da3aaa8f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35219c4e20d4f648d29ef4da1f3b305" is_pre_election: true
I20260928 15:06:27.811733 9959 raft_consensus.cc:2454] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3963f1aa44714c59b12676da3aaa8f30 in term 0.
I20260928 15:06:27.811679 9819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4410fa7a351349dda608ca35e727a39b" candidate_uuid: "3963f1aa44714c59b12676da3aaa8f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d3bc139e744aa8867d2956b2ec418c" is_pre_election: true
I20260928 15:06:27.811856 9819 raft_consensus.cc:2454] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3963f1aa44714c59b12676da3aaa8f30 in term 0.
I20260928 15:06:27.811950 10026 leader_election.cc:302] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963f1aa44714c59b12676da3aaa8f30, c35219c4e20d4f648d29ef4da1f3b305; no voters:
I20260928 15:06:27.812100 10155 raft_consensus.cc:2781] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:06:27.812180 10155 raft_consensus.cc:482] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:06:27.812209 10155 raft_consensus.cc:3037] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:27.813030 10155 raft_consensus.cc:504] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.813220 10155 leader_election.cc:288] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [CANDIDATE]: Term 1 election: Requested vote from peers c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573), 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619)
I20260928 15:06:27.813385 9819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4410fa7a351349dda608ca35e727a39b" candidate_uuid: "3963f1aa44714c59b12676da3aaa8f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d3bc139e744aa8867d2956b2ec418c"
I20260928 15:06:27.813381 9959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4410fa7a351349dda608ca35e727a39b" candidate_uuid: "3963f1aa44714c59b12676da3aaa8f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35219c4e20d4f648d29ef4da1f3b305"
I20260928 15:06:27.813464 9959 raft_consensus.cc:3037] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:27.813486 9819 raft_consensus.cc:3037] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:06:27.814191 9819 raft_consensus.cc:2454] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3963f1aa44714c59b12676da3aaa8f30 in term 1.
I20260928 15:06:27.814260 9959 raft_consensus.cc:2454] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3963f1aa44714c59b12676da3aaa8f30 in term 1.
I20260928 15:06:27.814435 10025 leader_election.cc:302] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d3bc139e744aa8867d2956b2ec418c, 3963f1aa44714c59b12676da3aaa8f30; no voters:
I20260928 15:06:27.814553 10155 raft_consensus.cc:2781] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:06:27.814729 10155 raft_consensus.cc:686] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 LEADER]: Becoming Leader. State: Replica: 3963f1aa44714c59b12676da3aaa8f30, State: Running, Role: LEADER
I20260928 15:06:27.814842 10155 consensus_queue.cc:237] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:27.815690 9676 catalog_manager.cc:5495] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "3963f1aa44714c59b12676da3aaa8f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } health_report { overall_health: UNKNOWN } } }
I20260928 15:06:27.821470 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.819132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.25.123.3:46197)
0928 15:06:27.819179 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:27.819182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:27.819198 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:06:27.819231 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:27.819231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:27.819247 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:27.819319 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:06:27.819465 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.820016 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.820782 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:27.821013 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:27.821085 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:27.821213 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:06:27.821308 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:06:27.821363 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:27.821396 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 15:06:27.821413 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:06:27.823408 9959 raft_consensus.cc:1261] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 1 FOLLOWER]: Refusing update from remote peer 3963f1aa44714c59b12676da3aaa8f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:06:27.823408 9819 raft_consensus.cc:1261] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Refusing update from remote peer 3963f1aa44714c59b12676da3aaa8f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:06:27.823841 10158 consensus_queue.cc:1035] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:06:27.823971 10155 consensus_queue.cc:1035] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:06:27.826618 10162 mvcc.cc:205] Tried to move back new op lower bound from 7334330318121385984 to 7334330318090285056. Current Snapshot: MvccSnapshot[applied={T|T < 7334330318121385984}]
I20260928 15:06:27.826947 10163 mvcc.cc:205] Tried to move back new op lower bound from 7334330318121385984 to 7334330318090285056. Current Snapshot: MvccSnapshot[applied={T|T < 7334330318121385984}]
I20260928 15:06:27.829711 9676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:27.829840 9676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:06:27.830708 9676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Sending DeleteTablet for 3 replicas of tablet 4410fa7a351349dda608ca35e727a39b
I20260928 15:06:27.831051 9939 tablet_service.cc:1514] Processing DeleteTablet for tablet 4410fa7a351349dda608ca35e727a39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:27 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36866
I20260928 15:06:27.831069 9796 tablet_service.cc:1514] Processing DeleteTablet for tablet 4410fa7a351349dda608ca35e727a39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:27 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:55456
I20260928 15:06:27.831218 10071 tablet_service.cc:1514] Processing DeleteTablet for tablet 4410fa7a351349dda608ca35e727a39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:06:27 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:49378
I20260928 15:06:27.831375 10171 tablet_replica.cc:330] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: stopping tablet replica
I20260928 15:06:27.831398 9874 tablet_replica.cc:330] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: stopping tablet replica
I20260928 15:06:27.831395 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 9714
I20260928 15:06:27.831455 9874 raft_consensus.cc:2229] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:27.831530 9874 raft_consensus.cc:2258] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:27.832206 10171 raft_consensus.cc:2229] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:06:27.832333 10171 raft_consensus.cc:2258] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:27.832863 10171 ts_tablet_manager.cc:1905] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:06:27.834796 10171 ts_tablet_manager.cc:1918] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:06:27.834893 10171 log.cc:1196] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/4410fa7a351349dda608ca35e727a39b
I20260928 15:06:27.835215 10171 ts_tablet_manager.cc:1939] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Deleting consensus metadata
I20260928 15:06:27.835815 9662 catalog_manager.cc:4841] TS c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573): tablet 4410fa7a351349dda608ca35e727a39b (table test-table [id=124718cbc9cd45c19663ff64c4ba02f8]) successfully deleted
I20260928 15:06:27.836536 10172 tablet_replica.cc:330] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: stopping tablet replica
I20260928 15:06:27.836666 10172 raft_consensus.cc:2229] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:06:27.836818 10172 raft_consensus.cc:2258] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:06:27.837277 10172 ts_tablet_manager.cc:1905] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 15:06:27.837473 9661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:46619 (error 108)
W20260928 15:06:27.838121 9872 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:06:27.837909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:46619 (local address 127.0.0.1:37080)
0928 15:06:27.837966 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.837978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:27.838038 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:27.838081 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:46619: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 15:06:27.838234 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 9869
W20260928 15:06:27.838249 9661 catalog_manager.cc:4584] TS 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4410fa7a351349dda608ca35e727a39b: Network error: Client connection negotiation failed: client connection to 127.25.123.1:46619: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 15:06:27.844265 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 10008
I20260928 15:06:27.849372 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 9643
I20260928 15:06:27.849459 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:27.849284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:46197 (local address 127.0.0.1:49400)
0928 15:06:27.849350 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:27.849418 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.3:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 15:06:47.855406 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37565
--webserver_interface=127.25.123.62
--webserver_port=36787
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:37565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:47.943358 10174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:47.943662 10174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:47.943691 10174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:47.944993 10174 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:06:47.945042 10174 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:06:47.945056 10174 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:06:47.945068 10174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:06:47.945081 10174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:47.945096 10174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:47.945107 10174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:06:47.945119 10174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:06:47.946710 10174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:37565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=36787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:47.946990 10174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:47.947247 10174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:47.953202 10182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:47.953235 10180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:47.953317 10174 server_base.cc:1034] running on GCE node
W20260928 15:06:47.953285 10179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:47.954200 10174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:47.954587 10174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:47.955787 10174 hybrid_clock.cc:648] HybridClock initialized: now 1790608007955761 us; error 69 us; skew 500 ppm
Sep 28 15:06:47 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608007, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:47.958819 10174 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:06:47.959302 10174 webserver.cc:460] Webserver started at http://127.25.123.62:36787/ using document root <none> and password file <none>
I20260928 15:06:47.959570 10174 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:47.959618 10174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:47.961193 10174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:06:47.962204 10189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:47.962450 10174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:06:47.962553 10174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "a5c510f895966c3c6528d3b805d0ee63"
server_key_iv: "19e83b5082378ca0c178d27b8b91c75d"
server_key_version: "encryptionkey@0"
I20260928 15:06:47.963002 10174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:47.992589 10174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:47.995194 10174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:47.995330 10174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:47.999470 10174 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:37565
I20260928 15:06:47.999548 10242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:37565 every 8 connection(s)
I20260928 15:06:47.999939 10174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:06:48.001585 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10174
I20260928 15:06:48.001783 10243 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:06:48.001937 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:46619
--local_ip_for_outbound_sockets=127.25.123.1
--tserver_master_addrs=127.25.123.62:37565
--webserver_port=43961
--webserver_interface=127.25.123.1
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:48.002624 10243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Bootstrap starting.
I20260928 15:06:48.012117 10243 log.cc:824] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:48.014751 10243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:48.015067 10243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Bootstrap complete.
I20260928 15:06:48.017737 10243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:48.018013 10243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea32c79a3d84c7991bcb2a7193cfcc5, State: Initialized, Role: FOLLOWER
I20260928 15:06:48.018229 10243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:48.018333 10243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:06:48.018388 10243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:06:48.018437 10243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:06:48.019455 10243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:48.019588 10243 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ea32c79a3d84c7991bcb2a7193cfcc5; no voters:
I20260928 15:06:48.019802 10243 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:06:48.019878 10247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:06:48.020020 10247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [term 3 LEADER]: Becoming Leader. State: Replica: 9ea32c79a3d84c7991bcb2a7193cfcc5, State: Running, Role: LEADER
I20260928 15:06:48.020099 10243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:06:48.020125 10247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } }
I20260928 15:06:48.020429 10247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea32c79a3d84c7991bcb2a7193cfcc5. Latest consensus state: current_term: 3 leader_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } } }
I20260928 15:06:48.020499 10247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:48.020692 10249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea32c79a3d84c7991bcb2a7193cfcc5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37565 } } }
I20260928 15:06:48.020781 10249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:06:48.020862 10253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:06:48.021953 10253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:06:48.022100 10253 catalog_manager.cc:1259] Loaded cluster ID: 445b375ede744372ac883e5657e03b5e
I20260928 15:06:48.022294 10253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:06:48.023142 10253 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:06:48.023340 10253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Loaded TSK: 0
I20260928 15:06:48.023399 10253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ea32c79a3d84c7991bcb2a7193cfcc5: Loaded TSK: 1
F20260928 15:06:48.023618 10253 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f985ce639ad google::LogMessage::Fail() at ??:0
@ 0x7f985ce6688a google::LogMessage::SendToLog() at ??:0
@ 0x7f985ce634de google::LogMessage::Flush() at ??:0
@ 0x7f985ce64a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f985f5d0104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f985f5cdd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9864dbd1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9864dbd48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9864dbd594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9864d83c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9864d8dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f985ed310d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f985ed26954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f985fb8a6db start_thread at ??:0
@ 0x7f985b96161f clone at ??:0
W20260928 15:06:48.112108 10245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:48.112325 10245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:48.112349 10245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:48.113581 10245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:48.113629 10245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:48.113656 10245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:06:48.115157 10245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:46619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=43961
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:48.115401 10245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:48.115660 10245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:48.122119 10285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:48.122134 10287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:48.122416 10245 server_base.cc:1034] running on GCE node
W20260928 15:06:48.122560 10284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:48.122761 10245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:48.123023 10245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:48.124200 10245 hybrid_clock.cc:648] HybridClock initialized: now 1790608008124188 us; error 30 us; skew 500 ppm
Sep 28 15:06:48 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608008, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:48.127190 10245 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:06:48.127882 10245 webserver.cc:460] Webserver started at http://127.25.123.1:43961/ using document root <none> and password file <none>
I20260928 15:06:48.128126 10245 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:48.128196 10245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:48.129776 10245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:48.130525 10294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:48.130767 10245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:48.130875 10245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "30d3bc139e744aa8867d2956b2ec418c"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "0166ebd19b8ee35fcc3b051ff770a801"
server_key_iv: "f38f0e70c53d4c43e4087d9697711ec6"
server_key_version: "encryptionkey@0"
I20260928 15:06:48.131367 10245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:48.206120 10245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:48.208863 10245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:48.209018 10245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:48.209292 10245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:48.209914 10301 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:06:48.212152 10245 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:06:48.212242 10245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:06:48.212285 10245 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:06:48.213084 10245 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:06:48.213161 10245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:06:48.213279 10301 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap starting.
I20260928 15:06:48.220846 10245 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:46619
I20260928 15:06:48.220995 10409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:46619 every 8 connection(s)
I20260928 15:06:48.221289 10245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:06:48.224725 10301 log.cc:824] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:48.227576 10301 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:48.227929 10301 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Bootstrap complete.
I20260928 15:06:48.229179 10301 ts_tablet_manager.cc:1397] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260928 15:06:48.229404 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10245
I20260928 15:06:48.229727 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:40573
--local_ip_for_outbound_sockets=127.25.123.2
--tserver_master_addrs=127.25.123.62:37565
--webserver_port=44823
--webserver_interface=127.25.123.2
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:06:48.230803 10301 raft_consensus.cc:348] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:48.231205 10301 raft_consensus.cc:729] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d3bc139e744aa8867d2956b2ec418c, State: Initialized, Role: FOLLOWER
I20260928 15:06:48.231364 10301 consensus_queue.cc:260] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:48.231700 10301 ts_tablet_manager.cc:1428] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260928 15:06:48.251325 10412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:06:48.222017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:36827)
0928 15:06:48.222199 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.222242 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:48.222723 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:48.251069 (+ 28346us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
W20260928 15:06:48.251781 10410 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:37565 (0 consecutive failures): Network error: Failed to ping master at 127.25.123.62:37565: Client connection negotiation failed: client connection to 127.25.123.62:37565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:06:48.252197 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.252071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:58033)
0928 15:06:48.252128 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.252173 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 15:06:48.252614 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.252517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:57539)
0928 15:06:48.252565 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.252592 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 15:06:48.252746 10410 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:06:48.328604 10414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:48.328835 10414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:48.328873 10414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:48.330039 10414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:48.330108 10414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:48.330153 10414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:06:48.331676 10414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:40573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=44823
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:48.331943 10414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:48.332172 10414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:48.337831 10423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:48.337853 10420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:48.337831 10421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:48.338080 10414 server_base.cc:1034] running on GCE node
I20260928 15:06:48.338290 10414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:48.338608 10414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:48.339787 10414 hybrid_clock.cc:648] HybridClock initialized: now 1790608008339784 us; error 41 us; skew 500 ppm
Sep 28 15:06:48 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608008, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:48.342646 10414 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:06:48.343156 10414 webserver.cc:460] Webserver started at http://127.25.123.2:44823/ using document root <none> and password file <none>
I20260928 15:06:48.343389 10414 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:48.343442 10414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:48.344867 10414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:48.345772 10430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:48.346098 10414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:06:48.346185 10414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "c35219c4e20d4f648d29ef4da1f3b305"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "8c6bc2cfba4e08ca94499ca39440f401"
server_key_iv: "ca4d958c38a1262e65fbbf3d3d5725b9"
server_key_version: "encryptionkey@0"
I20260928 15:06:48.346633 10414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:48.357728 10414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:48.360322 10414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:48.360606 10414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:48.360878 10414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:48.361212 10414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:06:48.361244 10414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:48.361269 10414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:06:48.361281 10414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:48.367520 10414 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:40573
I20260928 15:06:48.367605 10544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:40573 every 8 connection(s)
I20260928 15:06:48.367934 10414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:06:48.369108 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.368684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:48239)
0928 15:06:48.368876 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.368960 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
W20260928 15:06:48.369622 10545 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:37565 (0 consecutive failures): Network error: Failed to ping master at 127.25.123.62:37565: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
I20260928 15:06:48.369992 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.369863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:55523)
0928 15:06:48.369917 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.369946 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:06:48.370262 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.370199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:44247)
0928 15:06:48.370221 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.370240 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 15:06:48.370492 10545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:06:48.377166 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10414
I20260928 15:06:48.377532 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:46197
--local_ip_for_outbound_sockets=127.25.123.3
--tserver_master_addrs=127.25.123.62:37565
--webserver_port=46255
--webserver_interface=127.25.123.3
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:06:48.475054 10548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:06:48.475349 10548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:06:48.475396 10548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:06:48.476907 10548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:06:48.476982 10548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:06:48.477034 10548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:06:48.478734 10548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:46197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=46255
--tserver_master_addrs=127.25.123.62:37565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:06:48.479048 10548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:06:48.479310 10548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:06:48.486420 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:06:48.486505 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:48.486604 10548 server_base.cc:1034] running on GCE node
W20260928 15:06:48.486505 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:06:48.486910 10548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:06:48.487219 10548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:06:48.488437 10548 hybrid_clock.cc:648] HybridClock initialized: now 1790608008488377 us; error 27 us; skew 500 ppm
Sep 28 15:06:48 dist-test-slave-qnz0 krb5kdc[9091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608008, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:06:48.491863 10548 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:06:48.492461 10548 webserver.cc:460] Webserver started at http://127.25.123.3:46255/ using document root <none> and password file <none>
I20260928 15:06:48.492730 10548 fs_manager.cc:359] Metadata directory not provided
I20260928 15:06:48.492782 10548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:06:48.494297 10548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:06:48.495476 10563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:06:48.495941 10548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:06:48.496068 10548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "3963f1aa44714c59b12676da3aaa8f30"
format_stamp: "Formatted at 2026-09-28 15:06:06 on dist-test-slave-qnz0"
server_key: "4d62a211a77c4413bd6af3ed28c075e9"
server_key_iv: "63cddee9c3b5acd1f496d6239feebd12"
server_key_version: "encryptionkey@0"
I20260928 15:06:48.497097 10548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:06:48.520776 10548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:06:48.523221 10548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:06:48.523424 10548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:06:48.523703 10548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:06:48.524233 10570 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 15:06:48.526100 10548 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:06:48.526274 10548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 15:06:48.526319 10548 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:06:48.527030 10548 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:06:48.527092 10548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 15:06:48.527391 10570 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Bootstrap starting.
I20260928 15:06:48.530102 10570 log.cc:824] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:48.535066 10570 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:48.535135 10548 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:46197
I20260928 15:06:48.535370 10679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:46197 every 8 connection(s)
I20260928 15:06:48.535444 10570 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Bootstrap complete.
I20260928 15:06:48.535580 10548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:06:48.535610 10570 ts_tablet_manager.cc:1397] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260928 15:06:48.537146 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.536638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:60089)
0928 15:06:48.536854 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.536971 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
W20260928 15:06:48.537400 10680 heartbeater.cc:644] Failed to heartbeat to 127.25.123.62:37565 (0 consecutive failures): Network error: Failed to ping master at 127.25.123.62:37565: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
I20260928 15:06:48.537509 10570 raft_consensus.cc:348] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:48.537757 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.537645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:34911)
0928 15:06:48.537701 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.537730 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:06:48.537827 10570 raft_consensus.cc:729] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963f1aa44714c59b12676da3aaa8f30, State: Initialized, Role: FOLLOWER
I20260928 15:06:48.537976 10570 consensus_queue.cc:260] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:48.538223 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.538127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:43187)
0928 15:06:48.538169 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.538200 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 15:06:48.538249 10570 ts_tablet_manager.cc:1428] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30: Time spent starting tablet: real 0.003s user 0.002s sys 0.004s
W20260928 15:06:48.538398 10680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:06:48.539078 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:48.538843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:52887)
0928 15:06:48.538980 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:48.539026 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 15:06:48.544575 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10548
I20260928 15:06:49.253543 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:49.253199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:54071)
0928 15:06:49.253418 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:49.253470 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":67,"threads_started":1}
I20260928 15:06:49.371153 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:49.370904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:53533)
0928 15:06:49.371050 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:49.371093 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260928 15:06:49.540313 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:49.539994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:50907)
0928 15:06:49.540180 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:49.540237 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260928 15:06:49.991629 10688 raft_consensus.cc:482] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:06:49.991798 10688 raft_consensus.cc:504] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:49.992271 10688 leader_election.cc:288] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3:46197), c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573)
Sep 28 15:06:49 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608008, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.2@KRBTEST.COM
Sep 28 15:06:49 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608008, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.3@KRBTEST.COM
I20260928 15:06:50.006296 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:49.992427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:40573 (local address 127.25.123.1:37045)
0928 15:06:49.992565 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:49.992578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:49.992645 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:49.993576 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:49.993577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:49.993739 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:49.993791 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:49.993793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.994924 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.994925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:49.995810 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:49.995812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.996887 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.996889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:49.997021 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:49.998444 (+ 1423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:49.998465 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:50.001429 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:50.004088 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:50.004099 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:50.004102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:50.004497 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:50.004965 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:50.004966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:50.004967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:50.005124 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:50.005668 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:50.005675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:50.005830 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 15:06:50.005946 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:06:50.006181 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260928 15:06:50.006372 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:49.992565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:49079 (local address 127.25.123.3:46197)
0928 15:06:49.992754 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:49.992766 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:49.992790 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:06:49.992949 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:49.992951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:49.993006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:49.993166 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:49.993878 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.994998 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.995836 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.996858 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.997165 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:50.002229 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:50.002234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:50.002234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:50.002247 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:50.004282 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:50.004963 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:50.004966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:50.004966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:50.005093 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:50.005402 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:50.005403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:50.005404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:50.005710 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:50.005815 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:50.006110 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 15:06:50.006243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260928 15:06:50.006379 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:49.992560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:37045 (local address 127.25.123.2:40573)
0928 15:06:49.992754 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:49.992766 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:49.992791 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:06:49.993405 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:49.993407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:49.993444 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:49.993579 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:49.993917 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.994811 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.995939 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.996751 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.996910 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:50.001797 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:50.001803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:50.001804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:50.001821 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:50.003978 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:50.004586 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:50.004591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:50.004591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:50.004720 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:50.005232 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:50.005233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:50.005234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:50.005572 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:50.005664 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:50.006073 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 15:06:50.006247 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":57,"threads_started":1}
I20260928 15:06:50.006484 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:49.992482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:46197 (local address 127.25.123.1:49079)
0928 15:06:49.992596 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:49.992607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:49.992664 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:49.993220 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:49.993222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:49.993544 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:49.993734 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:49.993740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.995108 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.995109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:49.995728 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:49.995730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:49.997102 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:49.997103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:49.997180 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:49.998444 (+ 1264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:49.998461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:50.002039 (+ 3578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:50.004506 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:50.004507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:50.004507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:50.004771 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:50.005217 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:50.005218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:50.005219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:50.005296 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:50.005836 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:50.005837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:50.005893 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:06:50.006012 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:06:50.006281 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":127,"thread_start_us":58,"threads_started":1}
I20260928 15:06:50.007781 10633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4410fa7a351349dda608ca35e727a39b" candidate_uuid: "30d3bc139e744aa8867d2956b2ec418c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3963f1aa44714c59b12676da3aaa8f30" is_pre_election: true
W20260928 15:06:50.007949 10297 leader_election.cc:341] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573): Not found: Tablet not found: 4410fa7a351349dda608ca35e727a39b
I20260928 15:06:50.008050 10633 raft_consensus.cc:2454] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d3bc139e744aa8867d2956b2ec418c in term 1.
I20260928 15:06:50.008481 10296 leader_election.cc:302] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d3bc139e744aa8867d2956b2ec418c, 3963f1aa44714c59b12676da3aaa8f30; no voters: c35219c4e20d4f648d29ef4da1f3b305
I20260928 15:06:50.008664 10688 raft_consensus.cc:2781] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:06:50.008831 10688 raft_consensus.cc:482] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:06:50.008873 10688 raft_consensus.cc:3037] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:06:50.010653 10688 raft_consensus.cc:504] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:50.010897 10688 leader_election.cc:288] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [CANDIDATE]: Term 2 election: Requested vote from peers 3963f1aa44714c59b12676da3aaa8f30 (127.25.123.3:46197), c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573)
I20260928 15:06:50.011155 10633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4410fa7a351349dda608ca35e727a39b" candidate_uuid: "30d3bc139e744aa8867d2956b2ec418c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3963f1aa44714c59b12676da3aaa8f30"
W20260928 15:06:50.011274 10297 leader_election.cc:341] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573): Not found: Tablet not found: 4410fa7a351349dda608ca35e727a39b
I20260928 15:06:50.011332 10633 raft_consensus.cc:3037] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:06:50.012714 10633 raft_consensus.cc:2454] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d3bc139e744aa8867d2956b2ec418c in term 2.
I20260928 15:06:50.012967 10296 leader_election.cc:302] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d3bc139e744aa8867d2956b2ec418c, 3963f1aa44714c59b12676da3aaa8f30; no voters: c35219c4e20d4f648d29ef4da1f3b305
I20260928 15:06:50.013108 10688 raft_consensus.cc:2781] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:06:50.013278 10688 raft_consensus.cc:686] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [term 2 LEADER]: Becoming Leader. State: Replica: 30d3bc139e744aa8867d2956b2ec418c, State: Running, Role: LEADER
I20260928 15:06:50.013382 10688 consensus_queue.cc:237] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:50.013972 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.013794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:44951)
0928 15:06:50.013906 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:50.013945 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 15:06:50.372058 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.371757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:43891)
0928 15:06:50.371939 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:50.372000 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260928 15:06:50.391503 10633 raft_consensus.cc:1261] T 4410fa7a351349dda608ca35e727a39b P 3963f1aa44714c59b12676da3aaa8f30 [term 2 FOLLOWER]: Refusing update from remote peer 30d3bc139e744aa8867d2956b2ec418c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 15:06:50.391968 10688 consensus_queue.cc:1035] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 15:06:50.393271 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.393050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:53207)
0928 15:06:50.393154 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:50.393225 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":56,"threads_started":1}
I20260928 15:06:50.394309 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.394195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:57765)
0928 15:06:50.394241 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:50.394271 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 15:06:50.394523 10297 consensus_peers.cc:487] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c -> Peer c35219c4e20d4f648d29ef4da1f3b305 (127.25.123.2:40573): Couldn't send request to peer c35219c4e20d4f648d29ef4da1f3b305. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4410fa7a351349dda608ca35e727a39b. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:06:50.394711 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.394619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:59881)
0928 15:06:50.394658 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:50.394684 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 15:06:50.800925 10702 ts_tablet_manager.cc:927] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Initiating tablet copy from peer 30d3bc139e744aa8867d2956b2ec418c (127.25.123.1:46619)
I20260928 15:06:50.801149 10702 tablet_copy_client.cc:316] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: tablet copy: Beginning tablet copy session from remote peer at address 127.25.123.1:46619
Sep 28 15:06:50 dist-test-slave-qnz0 krb5kdc[9091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608008, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.1@KRBTEST.COM
I20260928 15:06:50.810662 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.801441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:46619 (local address 127.25.123.2:35585)
0928 15:06:50.801547 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:50.801578 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:06:50.801668 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:06:50.802025 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:06:50.802032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:06:50.802349 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:06:50.802424 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:50.802426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:50.803663 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:50.803664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:50.804293 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:06:50.804295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:06:50.805007 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:06:50.805008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:06:50.805093 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:50.805657 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:06:50.805662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:06:50.807203 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:06:50.809184 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:50.809192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:50.809194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:50.809440 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:50.809776 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:06:50.809777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:06:50.809777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:06:50.809845 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:06:50.810255 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:06:50.810266 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:06:50.810323 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:06:50.810469 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:06:50.810583 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 15:06:50.810757 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.801593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:35585 (local address 127.25.123.1:46619)
0928 15:06:50.801774 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:06:50.801778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:06:50.801789 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:06:50.801815 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:06:50.801816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:06:50.801831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:06:50.801945 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:06:50.802532 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:50.803563 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:50.804534 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:06:50.804896 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:06:50.804978 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:06:50.807330 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:06:50.807338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:06:50.807339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:06:50.807346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:06:50.809069 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:50.809544 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:50.809548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:50.809548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:50.809657 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:06:50.809960 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:06:50.809962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:06:50.809962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:06:50.810151 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:06:50.810237 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:06:50.810574 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 15:06:50.810660 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260928 15:06:50.811239 10384 tablet_copy_service.cc:140] P 30d3bc139e744aa8867d2956b2ec418c: Received BeginTabletCopySession request for tablet 4410fa7a351349dda608ca35e727a39b from peer c35219c4e20d4f648d29ef4da1f3b305 ({username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:35585)
I20260928 15:06:50.811326 10384 tablet_copy_service.cc:161] P 30d3bc139e744aa8867d2956b2ec418c: Beginning new tablet copy session on tablet 4410fa7a351349dda608ca35e727a39b from peer c35219c4e20d4f648d29ef4da1f3b305 at {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:35585: session id = c35219c4e20d4f648d29ef4da1f3b305-4410fa7a351349dda608ca35e727a39b
I20260928 15:06:50.811890 10384 tablet_copy_source_session.cc:205] T 4410fa7a351349dda608ca35e727a39b P 30d3bc139e744aa8867d2956b2ec418c: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:06:50.813153 10702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4410fa7a351349dda608ca35e727a39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:06:50.816140 10702 tablet_copy_client.cc:791] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: tablet copy: Starting download of 0 data blocks...
I20260928 15:06:50.816320 10702 tablet_copy_client.cc:655] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: tablet copy: Starting download of 1 WAL segments...
I20260928 15:06:50.817318 10702 tablet_copy_client.cc:531] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:06:50.819310 10702 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Bootstrap starting.
I20260928 15:06:50.820897 10702 log.cc:824] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Log is configured to *not* fsync() on all Append() calls
I20260928 15:06:50.823614 10702 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:06:50.823899 10702 tablet_bootstrap.cc:492] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Bootstrap complete.
I20260928 15:06:50.824067 10702 ts_tablet_manager.cc:1397] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 15:06:50.825340 10702 raft_consensus.cc:348] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:50.825516 10702 raft_consensus.cc:729] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35219c4e20d4f648d29ef4da1f3b305, State: Initialized, Role: FOLLOWER
I20260928 15:06:50.825664 10702 consensus_queue.cc:260] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963f1aa44714c59b12676da3aaa8f30" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 46197 } } peers { permanent_uuid: "c35219c4e20d4f648d29ef4da1f3b305" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 40573 } } peers { permanent_uuid: "30d3bc139e744aa8867d2956b2ec418c" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 46619 } }
I20260928 15:06:50.825942 10702 ts_tablet_manager.cc:1428] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260928 15:06:50.826350 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:50.826236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:59385)
0928 15:06:50.826291 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:50.826325 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 15:06:50.826349 10384 tablet_copy_service.cc:342] P 30d3bc139e744aa8867d2956b2ec418c: Request end of tablet copy session c35219c4e20d4f648d29ef4da1f3b305-4410fa7a351349dda608ca35e727a39b received from {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:35585
I20260928 15:06:50.826411 10384 tablet_copy_service.cc:433] P 30d3bc139e744aa8867d2956b2ec418c: ending tablet copy session c35219c4e20d4f648d29ef4da1f3b305-4410fa7a351349dda608ca35e727a39b on tablet 4410fa7a351349dda608ca35e727a39b with peer c35219c4e20d4f648d29ef4da1f3b305
I20260928 15:06:51.316371 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:51.316211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:53421)
0928 15:06:51.316291 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:51.316323 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":38,"threads_started":1}
I20260928 15:06:51.316522 10499 raft_consensus.cc:1203] T 4410fa7a351349dda608ca35e727a39b P c35219c4e20d4f648d29ef4da1f3b305 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 15:06:51.317126 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:51.316899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:57657)
0928 15:06:51.316972 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:51.317086 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:06:51.395356 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:51.395153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:40739)
0928 15:06:51.395269 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:51.395308 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260928 15:06:52.317348 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:52.317060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:53943)
0928 15:06:52.317246 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:52.317296 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260928 15:06:52.318032 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:52.317796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:33575)
0928 15:06:52.317962 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:52.317994 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260928 15:06:52.396332 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:52.396038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:42901)
0928 15:06:52.396234 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:52.396283 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":117,"threads_started":1}
I20260928 15:06:53.318495 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:53.318221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:36785)
0928 15:06:53.318370 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:53.318430 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260928 15:06:53.318771 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:53.318538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:33725)
0928 15:06:53.318650 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:53.318694 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260928 15:06:53.319144 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:53.319004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:58701)
0928 15:06:53.319082 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:53.319110 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:06:53.397101 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:53.396884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:39113)
0928 15:06:53.397012 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:53.397052 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260928 15:06:54.319873 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:54.319622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:55511)
0928 15:06:54.319783 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:54.319823 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260928 15:06:54.320039 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:54.319674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:37541)
0928 15:06:54.319898 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:54.319965 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260928 15:06:54.398347 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:54.397834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:45111)
0928 15:06:54.398242 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:54.398295 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":99,"threads_started":1}
I20260928 15:06:55.320734 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:55.320443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:36415)
0928 15:06:55.320604 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:55.320657 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260928 15:06:55.321105 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:55.320867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:44293)
0928 15:06:55.321019 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:55.321070 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260928 15:06:55.399314 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:55.399050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:56171)
0928 15:06:55.399212 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:55.399262 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260928 15:06:56.321620 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:56.321275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:52865)
0928 15:06:56.321484 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:56.321569 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":143,"threads_started":1}
I20260928 15:06:56.322054 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:56.321846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:46339)
0928 15:06:56.321963 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:56.322015 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260928 15:06:56.400352 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:56.400078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:55117)
0928 15:06:56.400243 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:56.400300 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":69,"threads_started":1}
I20260928 15:06:57.322746 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:57.322455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:48535)
0928 15:06:57.322633 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:57.322683 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260928 15:06:57.323009 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:57.322642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:60377)
0928 15:06:57.322787 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:57.322953 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260928 15:06:57.401252 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:57.400953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:49869)
0928 15:06:57.401153 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:57.401197 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260928 15:06:58.323644 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.323337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:53973)
0928 15:06:58.323544 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.323593 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":157,"threads_started":1}
I20260928 15:06:58.323818 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.323559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:34445)
0928 15:06:58.323684 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.323749 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260928 15:06:58.402292 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:58.402012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:54561)
0928 15:06:58.402178 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:58.402237 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260928 15:06:59.324793 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.324453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:57137)
0928 15:06:59.324693 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.324741 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":126,"threads_started":1}
I20260928 15:06:59.324793 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.324586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:59601)
0928 15:06:59.324714 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.324749 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":52,"threads_started":1}
I20260928 15:06:59.403265 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:06:59.402937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:39415)
0928 15:06:59.403156 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:06:59.403207 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260928 15:07:00.325745 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.325462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:60341)
0928 15:07:00.325642 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.325701 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260928 15:07:00.325739 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.325462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:48553)
0928 15:07:00.325642 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.325685 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260928 15:07:00.404031 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:00.403752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:42899)
0928 15:07:00.403935 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:00.403979 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":62,"threads_started":1}
I20260928 15:07:01.326696 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:01.326400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:44485)
0928 15:07:01.326589 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:01.326637 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260928 15:07:01.326742 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:01.326400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:51747)
0928 15:07:01.326647 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:01.326696 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":198,"threads_started":1}
I20260928 15:07:01.405019 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:01.404699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:37679)
0928 15:07:01.404856 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:01.404932 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260928 15:07:02.327555 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:02.327310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:49351)
0928 15:07:02.327481 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:02.327517 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260928 15:07:02.327590 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:02.327295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:39761)
0928 15:07:02.327441 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:02.327487 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260928 15:07:02.405812 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:02.405558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:38385)
0928 15:07:02.405706 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:02.405762 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260928 15:07:03.328500 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:03.328200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.2:57081)
0928 15:07:03.328343 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:03.328435 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260928 15:07:03.328820 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:03.328601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.1:53307)
0928 15:07:03.328746 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:03.328785 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 15:07:03.406741 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:03.406383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37565 (local address 127.25.123.3:57617)
0928 15:07:03.406642 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:03.406687 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.62:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":111,"threads_started":1}
I20260928 15:07:03.546763 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 10245
I20260928 15:07:03.551949 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 10414
I20260928 15:07:03.556895 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 10548
2026-09-28T15:07:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58681 ms)
[----------] 30 tests from SecurityITest (275486 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:03 dist-test-slave-qnz0 krb5kdc[10749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:03 dist-test-slave-qnz0 krb5kdc[10749](info): set up 2 sockets
Sep 28 15:07:03 dist-test-slave-qnz0 krb5kdc[10749](info): commencing operation
krb5kdc: starting...
W20260928 15:07:04.695292 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:04 dist-test-slave-qnz0 krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:04.824950 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:34711
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:34711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:04.910790 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:04.911054 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:04.911099 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:04.912328 10762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:04.912402 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:04.912464 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:04.912503 10762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:04.912532 10762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:04.914137 10762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:34711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:34711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:04.914434 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:04.914803 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:07:04.920723 10762 server_base.cc:1034] running on GCE node
W20260928 15:07:04.920809 10769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:04.920738 10771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:04.920738 10768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:04.921537 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:04.921821 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:04.922995 10762 hybrid_clock.cc:648] HybridClock initialized: now 1790608024922985 us; error 32 us; skew 500 ppm
Sep 28 15:07:04 dist-test-slave-qnz0 krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608024, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:04.925994 10762 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:04.926499 10762 webserver.cc:460] Webserver started at http://127.25.123.62:34629/ using document root <none> and password file <none>
I20260928 15:07:04.926746 10762 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:04.926815 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:04.926970 10762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:04.928090 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "5af582e79c6e48b8862054f46d997e88"
format_stamp: "Formatted at 2026-09-28 15:07:04 on dist-test-slave-qnz0"
server_key: "44d77018c50bccecc3d747f6cd4be02b"
server_key_iv: "cca5d037269e22e392547d45245a521f"
server_key_version: "encryptionkey@0"
I20260928 15:07:04.928543 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "5af582e79c6e48b8862054f46d997e88"
format_stamp: "Formatted at 2026-09-28 15:07:04 on dist-test-slave-qnz0"
server_key: "44d77018c50bccecc3d747f6cd4be02b"
server_key_iv: "cca5d037269e22e392547d45245a521f"
server_key_version: "encryptionkey@0"
I20260928 15:07:04.930166 10762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:04.931061 10778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:04.931290 10762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:04.931461 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "5af582e79c6e48b8862054f46d997e88"
format_stamp: "Formatted at 2026-09-28 15:07:04 on dist-test-slave-qnz0"
server_key: "44d77018c50bccecc3d747f6cd4be02b"
server_key_iv: "cca5d037269e22e392547d45245a521f"
server_key_version: "encryptionkey@0"
I20260928 15:07:04.931553 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:04.959230 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:04.962585 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:04.962785 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:04.967113 10762 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:34711
I20260928 15:07:04.967190 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:34711 every 8 connection(s)
I20260928 15:07:04.967562 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:04.968564 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:04.971024 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10762
I20260928 15:07:04.971170 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:04.971184 10831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88: Bootstrap starting.
I20260928 15:07:04.971392 26092 external_mini_cluster.cc:1428] Setting key 6efd5a32ef21e6c6e9fd6ddce761ca01
I20260928 15:07:04.972002 10831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:04.972357 10831 log.cc:824] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:04.973276 10831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88: No bootstrap required, opened a new log
I20260928 15:07:04.975235 10831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af582e79c6e48b8862054f46d997e88" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34711 } }
I20260928 15:07:04.975394 10831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:04.975417 10831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af582e79c6e48b8862054f46d997e88, State: Initialized, Role: FOLLOWER
I20260928 15:07:04.975569 10831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af582e79c6e48b8862054f46d997e88" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34711 } }
I20260928 15:07:04.975649 10831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:04.975680 10831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:04.975731 10831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:04.976528 10831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af582e79c6e48b8862054f46d997e88" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34711 } }
I20260928 15:07:04.976644 10831 leader_election.cc:302] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af582e79c6e48b8862054f46d997e88; no voters:
I20260928 15:07:04.976809 10831 leader_election.cc:288] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:04.976883 10836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:04.977043 10836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [term 1 LEADER]: Becoming Leader. State: Replica: 5af582e79c6e48b8862054f46d997e88, State: Running, Role: LEADER
I20260928 15:07:04.977077 10831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:04.977206 10836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af582e79c6e48b8862054f46d997e88" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34711 } }
I20260928 15:07:04.977517 10836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af582e79c6e48b8862054f46d997e88. Latest consensus state: current_term: 1 leader_uuid: "5af582e79c6e48b8862054f46d997e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af582e79c6e48b8862054f46d997e88" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34711 } } }
I20260928 15:07:04.977548 10837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af582e79c6e48b8862054f46d997e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af582e79c6e48b8862054f46d997e88" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34711 } } }
I20260928 15:07:04.977686 10837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:04.977895 10836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:04.978375 10842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:04.978943 10842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:07:04 dist-test-slave-qnz0 krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:04.980909 10842 catalog_manager.cc:1347] Generated new cluster ID: ef04712582aa4e87bf641416762980cd
I20260928 15:07:04.980978 10842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:04.985991 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:04.971987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.25.123.62:34711)
0928 15:07:04.972221 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:04.972229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:04.972268 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:07:04.972860 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:04.972868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:04.972919 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:04.973042 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:04.974073 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:04.975917 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:04.977053 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:04.977755 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:04.977876 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:04.981022 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:04.981025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:04.981026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:04.981037 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:04.983175 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:04.983646 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:04.983649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:04.983650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:04.983758 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:04.984104 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:04.984105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:04.984105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:04.984780 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:04.985123 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:04.985254 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 15:07:04.985672 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
I20260928 15:07:04.988828 10842 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:04.989398 10842 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:04.994820 10842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88: Generated new TSK 0
I20260928 15:07:04.995030 10842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:07:05.034566 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34711
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:05.134770 10857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:05.135038 10857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:05.135066 10857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:05.136562 10857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:05.136618 10857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:05.136652 10857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:07:05.138322 10857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:05.138602 10857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:05.138888 10857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:05.145105 10862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:05.145046 10863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:05.145054 10865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:05.145570 10857 server_base.cc:1034] running on GCE node
I20260928 15:07:05.145781 10857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:05.146020 10857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:05.147193 10857 hybrid_clock.cc:648] HybridClock initialized: now 1790608025147175 us; error 38 us; skew 500 ppm
Sep 28 15:07:05 dist-test-slave-qnz0 krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608025, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:05.150095 10857 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:07:05.150631 10857 webserver.cc:460] Webserver started at http://127.25.123.1:43511/ using document root <none> and password file <none>
I20260928 15:07:05.150911 10857 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:05.150961 10857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:05.151124 10857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:05.152253 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "8797ef5dc5894730a69077e63e4f5e03"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "e9a7db74615452c3f42605cc48d339df"
server_key_iv: "c6954f12c97ef4596f3c09a6d7ca14ce"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.152695 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "8797ef5dc5894730a69077e63e4f5e03"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "e9a7db74615452c3f42605cc48d339df"
server_key_iv: "c6954f12c97ef4596f3c09a6d7ca14ce"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.154278 10857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:05.155292 10872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.155575 10857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:05.155673 10857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "8797ef5dc5894730a69077e63e4f5e03"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "e9a7db74615452c3f42605cc48d339df"
server_key_iv: "c6954f12c97ef4596f3c09a6d7ca14ce"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.155731 10857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:05.172199 10857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:05.174738 10857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:05.174860 10857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:05.175060 10857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:05.175410 10857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:05.175557 10857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.175635 10857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:05.175660 10857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.182135 10857 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:41583
I20260928 15:07:05.182221 10986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:41583 every 8 connection(s)
I20260928 15:07:05.182585 10857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
Sep 28 15:07:05 dist-test-slave-qnz0 krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608025, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:05.191475 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10857
I20260928 15:07:05.191640 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:05.191860 26092 external_mini_cluster.cc:1428] Setting key c38df15e4b7e78e9de0c2fe662f913f5
I20260928 15:07:05.193460 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.183353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:34711 (local address 127.25.123.1:48721)
0928 15:07:05.183552 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.183597 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.184098 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.184366 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.184369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.185060 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:05.185272 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.185277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.186057 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.186058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.186850 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.186852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.187247 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.187248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.187360 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.187892 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:05.187900 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:05.189378 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:05.191430 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:05.191441 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:05.191445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:05.191748 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:05.191995 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:05.191996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:05.191997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:05.192167 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:05.192749 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:05.192758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:05.192971 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.193119 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.193272 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":118,"threads_started":1}
I20260928 15:07:05.193523 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.183405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:48721 (local address 127.25.123.62:34711)
0928 15:07:05.183475 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.183477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.183691 (+ 214us) server_negotiation.cc:407] Connection header received
0928 15:07:05.184196 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.184197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.184207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.184272 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:05.185400 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.185947 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.186940 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.187161 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.187226 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.189504 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:05.189506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:05.189506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:05.189513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:05.191296 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.191816 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.191817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.191818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.191882 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.192351 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.192353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.192353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.192661 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:05.192735 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.193186 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.193309 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 15:07:05.194123 10987 heartbeater.cc:347] Connected to a master server at 127.25.123.62:34711
I20260928 15:07:05.194214 10987 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:05.194425 10987 heartbeater.cc:511] Master 127.25.123.62:34711 requested a full tablet report, sending...
I20260928 15:07:05.194988 10795 ts_manager.cc:194] Registered new tserver with Master: 8797ef5dc5894730a69077e63e4f5e03 (127.25.123.1:41583)
I20260928 15:07:05.195786 10795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:48721
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:07:05.225596 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34711
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:05.312764 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:05.312992 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:05.313027 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:05.314160 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:05.314198 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:05.314224 10992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:07:05.315632 10992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:05.315831 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:05.316115 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:05.321473 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:05.321655 10992 server_base.cc:1034] running on GCE node
W20260928 15:07:05.321478 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:05.321478 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:05.321938 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:05.322223 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:05.323380 10992 hybrid_clock.cc:648] HybridClock initialized: now 1790608025323365 us; error 31 us; skew 500 ppm
Sep 28 15:07:05 dist-test-slave-qnz0 krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608025, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:05.326295 10992 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:07:05.326941 10992 webserver.cc:460] Webserver started at http://127.25.123.2:40789/ using document root <none> and password file <none>
I20260928 15:07:05.327222 10992 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:05.327298 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:05.327463 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:05.328656 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "df5a574fe15b9d11ef7aeac436bbf6c8"
server_key_iv: "897da7ded78a36d0b74e5b4dfdee95dd"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.329038 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "df5a574fe15b9d11ef7aeac436bbf6c8"
server_key_iv: "897da7ded78a36d0b74e5b4dfdee95dd"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.330518 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:05.331451 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.331697 10992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:05.331810 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "df5a574fe15b9d11ef7aeac436bbf6c8"
server_key_iv: "897da7ded78a36d0b74e5b4dfdee95dd"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.331899 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:05.363013 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:05.365763 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:05.365969 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:05.366250 10992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:05.366595 10992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:05.366652 10992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.366695 10992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:05.366724 10992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.373046 10992 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:43145
I20260928 15:07:05.373114 11121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:43145 every 8 connection(s)
I20260928 15:07:05.373472 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
Sep 28 15:07:05 dist-test-slave-qnz0 krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608025, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:05.381692 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 10992
I20260928 15:07:05.381888 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:05.382124 26092 external_mini_cluster.cc:1428] Setting key f5707d65cb71b73bc550c0ee1c91dce2
I20260928 15:07:05.384845 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.374347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:51971 (local address 127.25.123.62:34711)
0928 15:07:05.374504 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.374506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.374760 (+ 254us) server_negotiation.cc:407] Connection header received
0928 15:07:05.375321 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.375322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.375334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.375395 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:05.376381 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.376998 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.378121 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.378392 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.378485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.381013 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:05.381015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:05.381015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:05.381022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:05.383011 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.383633 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.383634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.383634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.383693 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.384014 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.384015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.384015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.384129 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:05.384185 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.384597 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.384722 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 15:07:05.384955 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.374234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:34711 (local address 127.25.123.2:51971)
0928 15:07:05.374624 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.374662 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.375211 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.375480 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.375483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.376073 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:05.376283 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.376287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.377099 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.377100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.378028 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.378031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.378494 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.378496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.378622 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.379205 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:05.379214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:05.380890 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:05.383117 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:05.383126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:05.383128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:05.383513 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:05.383770 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:05.383771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:05.383771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:05.383937 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:05.384198 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:05.384204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:05.384405 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.384547 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.384745 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":116,"threads_started":1}
I20260928 15:07:05.385596 11122 heartbeater.cc:347] Connected to a master server at 127.25.123.62:34711
I20260928 15:07:05.385725 11122 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:05.385933 11122 heartbeater.cc:511] Master 127.25.123.62:34711 requested a full tablet report, sending...
I20260928 15:07:05.386392 10795 ts_manager.cc:194] Registered new tserver with Master: 2e2500ff734c4e9b82fa6bbebab7a8b7 (127.25.123.2:43145)
I20260928 15:07:05.386793 10795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:51971
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:07:05.413034 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34711
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:05.503480 11127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:05.503744 11127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:05.503785 11127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:05.505075 11127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:05.505144 11127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:05.505194 11127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:07:05.506726 11127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:05.507028 11127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:05.507279 11127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:05.513381 11133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:05.513326 11132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:05.513326 11135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:05.513345 11127 server_base.cc:1034] running on GCE node
I20260928 15:07:05.513837 11127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:05.514086 11127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:05.515257 11127 hybrid_clock.cc:648] HybridClock initialized: now 1790608025515229 us; error 41 us; skew 500 ppm
Sep 28 15:07:05 dist-test-slave-qnz0 krb5kdc[10749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608025, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:05.518599 11127 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:07:05.519212 11127 webserver.cc:460] Webserver started at http://127.25.123.3:40459/ using document root <none> and password file <none>
I20260928 15:07:05.519665 11127 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:05.519734 11127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:05.519887 11127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:05.521201 11127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "0689f218c2ea4a8f9f6bc0b338967376"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "cadae487eab978fdca0eb586513cdd9a"
server_key_iv: "0492d0086c2949c1c2e57f9d7d2df399"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.521610 11127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "0689f218c2ea4a8f9f6bc0b338967376"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "cadae487eab978fdca0eb586513cdd9a"
server_key_iv: "0492d0086c2949c1c2e57f9d7d2df399"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.523183 11127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:05.524096 11142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.524509 11127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:05.524623 11127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "0689f218c2ea4a8f9f6bc0b338967376"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "cadae487eab978fdca0eb586513cdd9a"
server_key_iv: "0492d0086c2949c1c2e57f9d7d2df399"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.524719 11127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:05.542667 11127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:05.545405 11127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:05.545625 11127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:05.545898 11127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:05.546270 11127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:05.546337 11127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.546379 11127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:05.546435 11127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.553021 11127 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:33005
I20260928 15:07:05.553112 11255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:33005 every 8 connection(s)
I20260928 15:07:05.553468 11127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:05.559056 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 11127
I20260928 15:07:05.559146 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:05.559384 26092 external_mini_cluster.cc:1428] Setting key e0f0ceadc09352d7e0249fac7b16f7b0
Sep 28 15:07:05 dist-test-slave-qnz0 krb5kdc[10749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608025, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:05.563972 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.554267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:55795 (local address 127.25.123.62:34711)
0928 15:07:05.554356 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.554358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.554596 (+ 238us) server_negotiation.cc:407] Connection header received
0928 15:07:05.555109 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.555110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.555121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.555178 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:05.556190 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.556816 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.557848 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.558061 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.558186 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.561171 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:05.561172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:05.561173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:05.561181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:05.562528 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.562970 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.562971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.562971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.563031 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.563295 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.563296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.563296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.563416 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:05.563489 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.563793 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.563873 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:07:05.564215 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.554246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:34711 (local address 127.25.123.3:55795)
0928 15:07:05.554447 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.554485 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.555006 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.555245 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.555248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.555847 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:05.556069 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.556074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.556908 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.556909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.557767 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.557769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.558265 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.558266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.558398 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.558972 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:05.558980 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:05.561058 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:05.562623 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:05.562631 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:05.562633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:05.562874 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:05.563100 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:05.563101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:05.563102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:05.563223 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:05.563491 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:05.563498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:05.563650 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.563760 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.563934 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":114,"threads_started":1}
I20260928 15:07:05.564827 11256 heartbeater.cc:347] Connected to a master server at 127.25.123.62:34711
I20260928 15:07:05.564935 11256 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:05.565126 11256 heartbeater.cc:511] Master 127.25.123.62:34711 requested a full tablet report, sending...
I20260928 15:07:05.565546 10795 ts_manager.cc:194] Registered new tserver with Master: 0689f218c2ea4a8f9f6bc0b338967376 (127.25.123.3:33005)
I20260928 15:07:05.565891 10795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:55795
I20260928 15:07:05.571730 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:05.580639 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.573197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.25.123.62:34711)
0928 15:07:05.573242 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.573244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.573346 (+ 102us) server_negotiation.cc:407] Connection header received
0928 15:07:05.573458 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.573459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.573468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.573511 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:05.574122 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.574686 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.575423 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.575635 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.575700 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.577230 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:05.577231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:05.577232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:05.577258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:05.578917 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.579354 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.579355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.579355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.579419 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:05.579728 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:05.579730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:05.579730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:05.579904 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:05.580004 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.580365 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.580567 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:07:05.582508 10795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:05.584178 10795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:05.592202 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.587611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:33005 (local address 127.0.0.1:41786)
0928 15:07:05.588117 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.588128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.588179 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.588387 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.588387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.588409 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:05.588465 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.588472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.589493 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.589494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.590857 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.590860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.591850 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.591851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.591934 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.591950 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.592015 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.592098 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":85,"thread_start_us":70,"threads_started":1}
I20260928 15:07:05.592489 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.587646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.25.123.1:41583)
0928 15:07:05.588131 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.588136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.588148 (+ 12us) server_negotiation.cc:407] Connection header received
0928 15:07:05.588191 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.588193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.588211 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.588288 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:05.588539 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.589716 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.591340 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.591876 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.591947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.592006 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.592189 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.592237 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":111,"threads_started":1}
I20260928 15:07:05.592530 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.587373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:41583 (local address 127.0.0.1:54826)
0928 15:07:05.587773 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.587819 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.587915 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.588313 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.588316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.588319 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:05.588465 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.588468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.589841 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.589842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.591216 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.591218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.592040 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.592041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.592111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.592117 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.592173 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.592221 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":78,"thread_start_us":79,"threads_started":1}
I20260928 15:07:05.592582 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.587735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.25.123.3:33005)
0928 15:07:05.587913 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.587919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.588237 (+ 318us) server_negotiation.cc:407] Connection header received
0928 15:07:05.588279 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.588281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.588296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.588375 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:05.588604 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.589376 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.590958 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.591636 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.591709 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.592439 (+ 730us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.592464 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.592506 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260928 15:07:05.592583 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.587830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.25.123.2:43145)
0928 15:07:05.588472 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.588476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.588484 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:07:05.588524 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.588526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.588540 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.588621 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:05.588735 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.589786 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.591314 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.592050 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.592111 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.592159 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.592443 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.592492 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":68,"threads_started":1}
I20260928 15:07:05.593039 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.587373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43145 (local address 127.0.0.1:54390)
0928 15:07:05.587933 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.587944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.587999 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.588617 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.588618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.588620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:05.588663 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.588665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.589893 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.589893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.591215 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.591218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.592193 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.592194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.592257 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.592261 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.592313 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.592959 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":163,"thread_start_us":104,"threads_started":1}
I20260928 15:07:05.593545 11190 tablet_service.cc:1467] Processing CreateTablet for tablet dfa5c11051c64df29d2e545e11525445 (DEFAULT_TABLE table=test-table [id=3efb326c19424a02b4a954fe63a089eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:05.593725 10921 tablet_service.cc:1467] Processing CreateTablet for tablet dfa5c11051c64df29d2e545e11525445 (DEFAULT_TABLE table=test-table [id=3efb326c19424a02b4a954fe63a089eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:05.593729 11055 tablet_service.cc:1467] Processing CreateTablet for tablet dfa5c11051c64df29d2e545e11525445 (DEFAULT_TABLE table=test-table [id=3efb326c19424a02b4a954fe63a089eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:05.594084 10921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa5c11051c64df29d2e545e11525445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:05.594084 11055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa5c11051c64df29d2e545e11525445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:05.594162 11190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa5c11051c64df29d2e545e11525445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:05.597024 11274 tablet_bootstrap.cc:492] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Bootstrap starting.
I20260928 15:07:05.597539 11275 tablet_bootstrap.cc:492] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03: Bootstrap starting.
I20260928 15:07:05.597904 11276 tablet_bootstrap.cc:492] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Bootstrap starting.
I20260928 15:07:05.597967 11274 tablet_bootstrap.cc:654] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:05.598260 11275 tablet_bootstrap.cc:654] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:05.598479 11276 tablet_bootstrap.cc:654] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:05.598937 11274 log.cc:824] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:05.599141 11275 log.cc:824] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:05.599687 11276 log.cc:824] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:05.600014 11275 tablet_bootstrap.cc:492] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03: No bootstrap required, opened a new log
I20260928 15:07:05.600028 11274 tablet_bootstrap.cc:492] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: No bootstrap required, opened a new log
I20260928 15:07:05.600117 11274 ts_tablet_manager.cc:1397] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:05.600106 11275 ts_tablet_manager.cc:1397] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:05.600561 11276 tablet_bootstrap.cc:492] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: No bootstrap required, opened a new log
I20260928 15:07:05.600665 11276 ts_tablet_manager.cc:1397] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:05.601927 11275 raft_consensus.cc:348] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.602094 11275 raft_consensus.cc:374] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:05.602119 11275 raft_consensus.cc:729] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8797ef5dc5894730a69077e63e4f5e03, State: Initialized, Role: FOLLOWER
I20260928 15:07:05.602344 11275 consensus_queue.cc:260] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.602341 11274 raft_consensus.cc:348] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.602522 11274 raft_consensus.cc:374] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:05.602571 11274 raft_consensus.cc:729] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0689f218c2ea4a8f9f6bc0b338967376, State: Initialized, Role: FOLLOWER
I20260928 15:07:05.602684 11275 ts_tablet_manager.cc:1428] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:05.602824 10987 heartbeater.cc:503] Master 127.25.123.62:34711 was elected leader, sending a full tablet report...
I20260928 15:07:05.602824 11274 consensus_queue.cc:260] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.603034 11276 raft_consensus.cc:348] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.603314 11276 raft_consensus.cc:374] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:05.603360 11276 raft_consensus.cc:729] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2500ff734c4e9b82fa6bbebab7a8b7, State: Initialized, Role: FOLLOWER
I20260928 15:07:05.603483 11276 consensus_queue.cc:260] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.603628 11274 ts_tablet_manager.cc:1428] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:05.603824 11256 heartbeater.cc:503] Master 127.25.123.62:34711 was elected leader, sending a full tablet report...
I20260928 15:07:05.604156 11276 ts_tablet_manager.cc:1428] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:05.604195 11122 heartbeater.cc:503] Master 127.25.123.62:34711 was elected leader, sending a full tablet report...
W20260928 15:07:05.624150 11123 tablet.cc:2369] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:05.650779 11280 raft_consensus.cc:482] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:05.650921 11280 raft_consensus.cc:504] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.651252 11280 leader_election.cc:288] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e2500ff734c4e9b82fa6bbebab7a8b7 (127.25.123.2:43145), 0689f218c2ea4a8f9f6bc0b338967376 (127.25.123.3:33005)
I20260928 15:07:05.654546 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.651742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:46771 (local address 127.25.123.2:43145)
0928 15:07:05.651814 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.651816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.651827 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:05.651864 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.651865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.651875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.651962 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:05.652179 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.652769 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.653798 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.654174 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.654231 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.654264 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.654428 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.654469 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:07:05.654556 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.651574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:43145 (local address 127.25.123.1:46771)
0928 15:07:05.651646 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.651659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.651713 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.652006 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.652008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.652010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:05.652088 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.652091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.652860 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.652862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.653697 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.653699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.654261 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.654262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.654349 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.654352 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.654395 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.654462 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":37}
I20260928 15:07:05.654565 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.651915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:45275 (local address 127.25.123.3:33005)
0928 15:07:05.651984 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.651986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.651995 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:05.652022 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.652022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.652031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.652094 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:05.652274 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.652782 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.653795 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.654179 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.654270 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.654279 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.654462 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.654505 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:07:05.654563 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.651708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:33005 (local address 127.25.123.1:45275)
0928 15:07:05.651912 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.651923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:05.651973 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:05.652102 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:05.652103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:05.652105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:05.652197 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.652199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.652912 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.652914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.653701 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:05.653703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.654290 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.654292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:05.654358 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.654361 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:05.654435 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:07:05.654473 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":1,"thread_start_us":133,"threads_started":1}
I20260928 15:07:05.655035 11076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5c11051c64df29d2e545e11525445" candidate_uuid: "8797ef5dc5894730a69077e63e4f5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" is_pre_election: true
I20260928 15:07:05.655232 11076 raft_consensus.cc:2454] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8797ef5dc5894730a69077e63e4f5e03 in term 0.
I20260928 15:07:05.655418 11210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5c11051c64df29d2e545e11525445" candidate_uuid: "8797ef5dc5894730a69077e63e4f5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0689f218c2ea4a8f9f6bc0b338967376" is_pre_election: true
I20260928 15:07:05.655476 10875 leader_election.cc:302] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2500ff734c4e9b82fa6bbebab7a8b7, 8797ef5dc5894730a69077e63e4f5e03; no voters:
I20260928 15:07:05.655534 11210 raft_consensus.cc:2454] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8797ef5dc5894730a69077e63e4f5e03 in term 0.
I20260928 15:07:05.655642 11280 raft_consensus.cc:2781] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:05.655707 11280 raft_consensus.cc:482] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:05.655735 11280 raft_consensus.cc:3037] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:05.656498 11280 raft_consensus.cc:504] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.656608 11280 leader_election.cc:288] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [CANDIDATE]: Term 1 election: Requested vote from peers 2e2500ff734c4e9b82fa6bbebab7a8b7 (127.25.123.2:43145), 0689f218c2ea4a8f9f6bc0b338967376 (127.25.123.3:33005)
I20260928 15:07:05.656821 11210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5c11051c64df29d2e545e11525445" candidate_uuid: "8797ef5dc5894730a69077e63e4f5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0689f218c2ea4a8f9f6bc0b338967376"
I20260928 15:07:05.656847 11076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5c11051c64df29d2e545e11525445" candidate_uuid: "8797ef5dc5894730a69077e63e4f5e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7"
I20260928 15:07:05.656924 11076 raft_consensus.cc:3037] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:05.656934 11210 raft_consensus.cc:3037] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:05.657665 11076 raft_consensus.cc:2454] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8797ef5dc5894730a69077e63e4f5e03 in term 1.
I20260928 15:07:05.657789 11210 raft_consensus.cc:2454] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8797ef5dc5894730a69077e63e4f5e03 in term 1.
I20260928 15:07:05.657842 10875 leader_election.cc:302] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2500ff734c4e9b82fa6bbebab7a8b7, 8797ef5dc5894730a69077e63e4f5e03; no voters:
I20260928 15:07:05.657951 11280 raft_consensus.cc:2781] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:05.658116 11280 raft_consensus.cc:686] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [term 1 LEADER]: Becoming Leader. State: Replica: 8797ef5dc5894730a69077e63e4f5e03, State: Running, Role: LEADER
I20260928 15:07:05.658223 11280 consensus_queue.cc:237] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } }
I20260928 15:07:05.659163 10794 catalog_manager.cc:5495] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8797ef5dc5894730a69077e63e4f5e03 (127.25.123.1). New cstate: current_term: 1 leader_uuid: "8797ef5dc5894730a69077e63e4f5e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8797ef5dc5894730a69077e63e4f5e03" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 41583 } health_report { overall_health: HEALTHY } } }
I20260928 15:07:05.674789 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.672150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.25.123.1:41583)
0928 15:07:05.672199 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.672206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.672215 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:05.672271 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.672272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.672289 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.672351 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:05.672541 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.673090 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.673925 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.674148 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.674220 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.674422 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:05.674542 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:05.674618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.674671 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.674694 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:07:05.677045 11210 raft_consensus.cc:1261] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 1 FOLLOWER]: Refusing update from remote peer 8797ef5dc5894730a69077e63e4f5e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:05.677044 11076 raft_consensus.cc:1261] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 1 FOLLOWER]: Refusing update from remote peer 8797ef5dc5894730a69077e63e4f5e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:05.677515 11284 consensus_queue.cc:1035] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0689f218c2ea4a8f9f6bc0b338967376" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:05.677704 11280 consensus_queue.cc:1035] T dfa5c11051c64df29d2e545e11525445 P 8797ef5dc5894730a69077e63e4f5e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2500ff734c4e9b82fa6bbebab7a8b7" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 43145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:05.680289 11287 mvcc.cc:205] Tried to move back new op lower bound from 7334330473169805312 to 7334330473096896512. Current Snapshot: MvccSnapshot[applied={T|T < 7334330473169805312}]
I20260928 15:07:05.680326 11289 mvcc.cc:205] Tried to move back new op lower bound from 7334330473169805312 to 7334330473096896512. Current Snapshot: MvccSnapshot[applied={T|T < 7334330473169805312}]
I20260928 15:07:05.680773 11288 mvcc.cc:205] Tried to move back new op lower bound from 7334330473169805312 to 7334330473096896512. Current Snapshot: MvccSnapshot[applied={T|T < 7334330473169805312}]
I20260928 15:07:05.683087 10795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:05.683205 10795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:05.683966 10795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5af582e79c6e48b8862054f46d997e88: Sending DeleteTablet for 3 replicas of tablet dfa5c11051c64df29d2e545e11525445
I20260928 15:07:05.684305 11055 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa5c11051c64df29d2e545e11525445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:05 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:54390
I20260928 15:07:05.684309 10921 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa5c11051c64df29d2e545e11525445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:05 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:54826
I20260928 15:07:05.684455 11190 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa5c11051c64df29d2e545e11525445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:05 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:41786
I20260928 15:07:05.684696 11298 tablet_replica.cc:330] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: stopping tablet replica
I20260928 15:07:05.684779 11296 tablet_replica.cc:330] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: stopping tablet replica
I20260928 15:07:05.684816 11298 raft_consensus.cc:2229] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:05.684952 11298 raft_consensus.cc:2258] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:05.684959 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 10857
I20260928 15:07:05.686153 11296 raft_consensus.cc:2229] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:05.686334 11296 raft_consensus.cc:2258] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:05.686332 11298 ts_tablet_manager.cc:1905] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:05.686887 11296 ts_tablet_manager.cc:1905] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:05.688803 11296 ts_tablet_manager.cc:1918] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:05.688803 11298 ts_tablet_manager.cc:1918] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:05.688895 11298 log.cc:1196] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/dfa5c11051c64df29d2e545e11525445
I20260928 15:07:05.688896 11296 log.cc:1196] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/dfa5c11051c64df29d2e545e11525445
I20260928 15:07:05.689195 11296 ts_tablet_manager.cc:1939] T dfa5c11051c64df29d2e545e11525445 P 2e2500ff734c4e9b82fa6bbebab7a8b7: Deleting consensus metadata
I20260928 15:07:05.689195 11298 ts_tablet_manager.cc:1939] T dfa5c11051c64df29d2e545e11525445 P 0689f218c2ea4a8f9f6bc0b338967376: Deleting consensus metadata
I20260928 15:07:05.689649 10781 catalog_manager.cc:4841] TS 2e2500ff734c4e9b82fa6bbebab7a8b7 (127.25.123.2:43145): tablet dfa5c11051c64df29d2e545e11525445 (table test-table [id=3efb326c19424a02b4a954fe63a089eb]) successfully deleted
I20260928 15:07:05.689790 10781 catalog_manager.cc:4841] TS 0689f218c2ea4a8f9f6bc0b338967376 (127.25.123.3:33005): tablet dfa5c11051c64df29d2e545e11525445 (table test-table [id=3efb326c19424a02b4a954fe63a089eb]) successfully deleted
W20260928 15:07:05.691697 10780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:41583 (error 108)
I20260928 15:07:05.692297 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 10992
I20260928 15:07:05.692324 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.692176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:41583 (local address 127.0.0.1:54856)
0928 15:07:05.692240 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:05.692291 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 15:07:05.692485 10780 catalog_manager.cc:4584] TS 8797ef5dc5894730a69077e63e4f5e03 (127.25.123.1:41583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfa5c11051c64df29d2e545e11525445: Network error: Client connection negotiation failed: client connection to 127.25.123.1:41583: connect: Connection refused (error 111)
I20260928 15:07:05.698065 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 11127
I20260928 15:07:05.705559 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 10762
2026-09-28T15:07:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2147 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T15:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:05Z Disabled control of system clock
I20260928 15:07:05.734745 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:41089
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:41089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:05.821456 11303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:05.821720 11303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:05.821745 11303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:05.822938 11303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:05.822986 11303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:05.823000 11303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:05.823014 11303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:05.823025 11303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:05.824421 11303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:41089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:41089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:05.824676 11303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:05.824944 11303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:05.830384 11309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:05.830406 11308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:05.830390 11311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:05.830652 11303 server_base.cc:1034] running on GCE node
I20260928 15:07:05.831153 11303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:05.831463 11303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:05.832639 11303 hybrid_clock.cc:648] HybridClock initialized: now 1790608025832615 us; error 47 us; skew 500 ppm
I20260928 15:07:05.834115 11303 webserver.cc:460] Webserver started at http://127.25.123.62:45253/ using document root <none> and password file <none>
I20260928 15:07:05.834425 11303 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:05.834515 11303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:05.834678 11303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:05.835836 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "92b5be458a2242678285c4d0d9998d46"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "340c864a4ca3c998e6fb5f728b2352a7"
server_key_iv: "884f39aba4cbfb92405b9506fb540af6"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.836220 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "92b5be458a2242678285c4d0d9998d46"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "340c864a4ca3c998e6fb5f728b2352a7"
server_key_iv: "884f39aba4cbfb92405b9506fb540af6"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.837725 11303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:05.839473 11317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:05.839787 11303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:07:05.839917 11303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "92b5be458a2242678285c4d0d9998d46"
format_stamp: "Formatted at 2026-09-28 15:07:05 on dist-test-slave-qnz0"
server_key: "340c864a4ca3c998e6fb5f728b2352a7"
server_key_iv: "884f39aba4cbfb92405b9506fb540af6"
server_key_version: "encryptionkey@0"
I20260928 15:07:05.840024 11303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:05.847741 11303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:05.848126 11303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:05.848296 11303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:05.852556 11303 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:41089
I20260928 15:07:05.852632 11369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:41089 every 8 connection(s)
I20260928 15:07:05.852995 11303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:05.854025 11371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:05.856571 11371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46: Bootstrap starting.
I20260928 15:07:05.857215 11371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:05.857525 11371 log.cc:824] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:05.858315 11371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46: No bootstrap required, opened a new log
I20260928 15:07:05.859769 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 11303
I20260928 15:07:05.859912 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:05.860113 26092 external_mini_cluster.cc:1428] Setting key 1e26ac606689e3b2ccd17558a109788d
I20260928 15:07:05.860088 11371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b5be458a2242678285c4d0d9998d46" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 41089 } }
I20260928 15:07:05.860271 11371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:05.860317 11371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5be458a2242678285c4d0d9998d46, State: Initialized, Role: FOLLOWER
I20260928 15:07:05.860503 11371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b5be458a2242678285c4d0d9998d46" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 41089 } }
I20260928 15:07:05.860649 11371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:05.860713 11371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:05.860773 11371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:05.861388 11371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b5be458a2242678285c4d0d9998d46" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 41089 } }
I20260928 15:07:05.861527 11371 leader_election.cc:302] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b5be458a2242678285c4d0d9998d46; no voters:
I20260928 15:07:05.861752 11371 leader_election.cc:288] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:05.861783 11376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:05.861891 11376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [term 1 LEADER]: Becoming Leader. State: Replica: 92b5be458a2242678285c4d0d9998d46, State: Running, Role: LEADER
I20260928 15:07:05.862035 11371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:05.862030 11376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b5be458a2242678285c4d0d9998d46" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 41089 } }
I20260928 15:07:05.863974 11378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b5be458a2242678285c4d0d9998d46. Latest consensus state: current_term: 1 leader_uuid: "92b5be458a2242678285c4d0d9998d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b5be458a2242678285c4d0d9998d46" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 41089 } } }
I20260928 15:07:05.864070 11378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:05.863981 11377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92b5be458a2242678285c4d0d9998d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b5be458a2242678285c4d0d9998d46" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 41089 } } }
I20260928 15:07:05.864828 11377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46 [sys.catalog]: This master's current role is: LEADER
W20260928 15:07:05.864514 11390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:05.864918 11390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:05.865005 11391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:05.865259 11391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:05.866750 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:05.860700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.25.123.62:41089)
0928 15:07:05.860951 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:05.860960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:05.860995 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:07:05.861099 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:05.861104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:05.861177 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:05.861323 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:05.861538 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.863118 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.864308 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:05.864945 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:05.865078 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:05.865975 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:05.865979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:05.865979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:05.866187 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:05.866227 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:05.866242 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:05.866309 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:05.866427 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 15:07:05.866538 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
I20260928 15:07:05.867012 11391 catalog_manager.cc:1347] Generated new cluster ID: bbfd2b035d444496afe79630119025a2
I20260928 15:07:05.867100 11391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:05.908202 11391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:05.909026 11391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:05.917517 11391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46: Generated new TSK 0
I20260928 15:07:05.917776 11391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:05.920086 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:41089
--builtin_ntp_servers=127.25.123.20:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:06.013953 11395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:06.014189 11395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:06.014213 11395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:06.015602 11395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:06.015650 11395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:06.015679 11395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:07:06.017234 11395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:41089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:06.017526 11395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:06.017871 11395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:06.024287 11400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:06.024292 11403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:06.024314 11401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:06.024902 11395 server_base.cc:1034] running on GCE node
I20260928 15:07:06.025195 11395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:06.025556 11395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:06.026731 11395 hybrid_clock.cc:648] HybridClock initialized: now 1790608026026703 us; error 39 us; skew 500 ppm
I20260928 15:07:06.028193 11395 webserver.cc:460] Webserver started at http://127.25.123.1:34007/ using document root <none> and password file <none>
I20260928 15:07:06.028599 11395 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:06.028683 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:06.028838 11395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:06.030148 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "c067b0db200c4447a72b06750cc9ab83"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "8cfc442da09741fb5f2c1830f4f18ee6"
server_key_iv: "5ca162cd58714bbe7b62a62e03f8dae1"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.030596 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "c067b0db200c4447a72b06750cc9ab83"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "8cfc442da09741fb5f2c1830f4f18ee6"
server_key_iv: "5ca162cd58714bbe7b62a62e03f8dae1"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.032198 11395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:06.033357 11409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.033653 11395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:06.033772 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "c067b0db200c4447a72b06750cc9ab83"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "8cfc442da09741fb5f2c1830f4f18ee6"
server_key_iv: "5ca162cd58714bbe7b62a62e03f8dae1"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.033878 11395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:06.070590 11395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:06.070968 11395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:06.071149 11395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:06.071410 11395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:06.071768 11395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:06.071805 11395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.071862 11395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:06.071895 11395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.079427 11395 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:37511
I20260928 15:07:06.079483 11523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:37511 every 8 connection(s)
I20260928 15:07:06.079897 11395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:06.085742 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.080868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:38669 (local address 127.25.123.62:41089)
0928 15:07:06.080993 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.080996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.081204 (+ 208us) server_negotiation.cc:407] Connection header received
0928 15:07:06.081785 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.081786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.081804 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.081858 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:06.082595 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.083119 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.084139 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.084355 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.084487 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.085436 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:06.085438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:06.085438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:06.085446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:06.085461 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:06.085466 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:06.085522 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.085620 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.085655 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:07:06.085999 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.080820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:41089 (local address 127.25.123.1:38669)
0928 15:07:06.081083 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.081118 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.081664 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.081926 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.081931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.081942 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:06.082484 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.082488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.083222 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.083223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.084023 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.084025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.084519 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.084520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.084652 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.085271 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:06.085271 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:06.085300 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:06.085324 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:06.085522 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:06.085528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:06.085532 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.085621 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.085728 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260928 15:07:06.086539 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 11395
I20260928 15:07:06.086591 11524 heartbeater.cc:347] Connected to a master server at 127.25.123.62:41089
I20260928 15:07:06.086658 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:06.086736 11524 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:06.086892 26092 external_mini_cluster.cc:1428] Setting key a6d66e078abd6bd17506321adedba4cc
I20260928 15:07:06.086956 11524 heartbeater.cc:511] Master 127.25.123.62:41089 requested a full tablet report, sending...
I20260928 15:07:06.087550 11333 ts_manager.cc:194] Registered new tserver with Master: c067b0db200c4447a72b06750cc9ab83 (127.25.123.1:37511)
I20260928 15:07:06.088027 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:41089
--builtin_ntp_servers=127.25.123.20:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:06.088541 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.1:38669
W20260928 15:07:06.173882 11527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:06.174137 11527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:06.174175 11527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:06.175360 11527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:06.175429 11527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:06.175475 11527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:07:06.176954 11527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:41089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:06.177228 11527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:06.177542 11527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:07:06.182951 11527 server_base.cc:1034] running on GCE node
W20260928 15:07:06.183099 11533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:06.182920 11532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:06.182945 11535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:06.183445 11527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:06.183707 11527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:06.184868 11527 hybrid_clock.cc:648] HybridClock initialized: now 1790608026184857 us; error 28 us; skew 500 ppm
I20260928 15:07:06.186255 11527 webserver.cc:460] Webserver started at http://127.25.123.2:42991/ using document root <none> and password file <none>
I20260928 15:07:06.186524 11527 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:06.186574 11527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:06.186695 11527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:06.187813 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "52ccc7253e5a4219a15ccd5e937108c8"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "9052cd3daa78f404fda1a4f1f3662fcd"
server_key_iv: "7dae5f62c40073e80c755f7383d3c0fa"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.188135 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "52ccc7253e5a4219a15ccd5e937108c8"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "9052cd3daa78f404fda1a4f1f3662fcd"
server_key_iv: "7dae5f62c40073e80c755f7383d3c0fa"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.189648 11527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:06.190546 11541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.190747 11527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:06.190852 11527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "52ccc7253e5a4219a15ccd5e937108c8"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "9052cd3daa78f404fda1a4f1f3662fcd"
server_key_iv: "7dae5f62c40073e80c755f7383d3c0fa"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.190940 11527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:06.219062 11527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:06.219482 11527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:06.219655 11527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:06.219875 11527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:06.220227 11527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:06.220284 11527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.220337 11527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:06.220368 11527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.226615 11527 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:34813
I20260928 15:07:06.226693 11655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:34813 every 8 connection(s)
I20260928 15:07:06.227074 11527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:06.232925 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.227893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:52609 (local address 127.25.123.62:41089)
0928 15:07:06.227973 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.227975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.228225 (+ 250us) server_negotiation.cc:407] Connection header received
0928 15:07:06.228856 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.228858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.228869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.228930 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:06.229652 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.230152 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.231208 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.231397 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.231458 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.232586 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:06.232587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:06.232587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:06.232593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:06.232608 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:06.232612 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:06.232661 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.232835 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.232861 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 15:07:06.233186 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.227877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:41089 (local address 127.25.123.2:52609)
0928 15:07:06.228100 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.228135 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.228711 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.228990 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.228993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.229002 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:06.229544 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.229549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.230248 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.230248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.231120 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.231123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.231503 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.231504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.231664 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.232252 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:06.232252 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:06.232283 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:06.232308 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:06.232718 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:06.232724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:06.232727 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.232807 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.232903 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260928 15:07:06.233727 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 11527
I20260928 15:07:06.233772 11656 heartbeater.cc:347] Connected to a master server at 127.25.123.62:41089
I20260928 15:07:06.233855 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:06.233906 11656 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:06.234064 26092 external_mini_cluster.cc:1428] Setting key ba78e7178052de2ed78b8edbd94c05e7
I20260928 15:07:06.234136 11656 heartbeater.cc:511] Master 127.25.123.62:41089 requested a full tablet report, sending...
I20260928 15:07:06.234607 11334 ts_manager.cc:194] Registered new tserver with Master: 52ccc7253e5a4219a15ccd5e937108c8 (127.25.123.2:34813)
I20260928 15:07:06.235083 11334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.2:52609
I20260928 15:07:06.235226 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:41089
--builtin_ntp_servers=127.25.123.20:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:06.320730 11659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:06.320951 11659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:06.320974 11659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:06.322052 11659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:06.322086 11659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:06.322111 11659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:07:06.323379 11659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:41089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:06.323575 11659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:06.323839 11659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:06.329368 11667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:06.329478 11665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:06.329356 11664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:06.329603 11659 server_base.cc:1034] running on GCE node
I20260928 15:07:06.329891 11659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:06.330193 11659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:06.331399 11659 hybrid_clock.cc:648] HybridClock initialized: now 1790608026331374 us; error 34 us; skew 500 ppm
I20260928 15:07:06.332917 11659 webserver.cc:460] Webserver started at http://127.25.123.3:42933/ using document root <none> and password file <none>
I20260928 15:07:06.333230 11659 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:06.333307 11659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:06.333474 11659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:06.334555 11659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "84d5898785c140a497793fbec2725734"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "de211f6c9ea08acaae37587fc1e0ea74"
server_key_iv: "6e728383fecf14f944d3ad673ba905bc"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.334930 11659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "84d5898785c140a497793fbec2725734"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "de211f6c9ea08acaae37587fc1e0ea74"
server_key_iv: "6e728383fecf14f944d3ad673ba905bc"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.336354 11659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:07:06.337318 11673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.337507 11659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:06.337615 11659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "84d5898785c140a497793fbec2725734"
format_stamp: "Formatted at 2026-09-28 15:07:06 on dist-test-slave-qnz0"
server_key: "de211f6c9ea08acaae37587fc1e0ea74"
server_key_iv: "6e728383fecf14f944d3ad673ba905bc"
server_key_version: "encryptionkey@0"
I20260928 15:07:06.337703 11659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:06.361440 11659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:06.361832 11659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:06.362053 11659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:06.362354 11659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:06.362736 11659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:06.362772 11659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.362795 11659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:06.362843 11659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:06.368927 11659 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:36953
I20260928 15:07:06.368975 11786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:36953 every 8 connection(s)
I20260928 15:07:06.369347 11659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:06.370858 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 11659
I20260928 15:07:06.370954 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:06.371165 26092 external_mini_cluster.cc:1428] Setting key f40b3546b48aa0e0841d7255ebcac05e
I20260928 15:07:06.375321 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.370191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:49419 (local address 127.25.123.62:41089)
0928 15:07:06.370266 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.370269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.370516 (+ 247us) server_negotiation.cc:407] Connection header received
0928 15:07:06.371125 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.371126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.371141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.371202 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:06.372040 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.372669 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.373846 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.374030 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.374086 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.374988 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:06.374989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:06.374990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:06.374994 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:06.375007 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:06.375012 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:06.375065 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.375154 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.375224 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:07:06.375408 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.370151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:41089 (local address 127.25.123.3:49419)
0928 15:07:06.370382 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.370423 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.370989 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.371270 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.371274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.371282 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:06.371864 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.371869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.372773 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.372774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.373672 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.373674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.374110 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.374111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.374244 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.374831 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:06.374832 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:06.374862 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:06.374894 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:06.375080 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:06.375084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:06.375087 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.375141 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.375225 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260928 15:07:06.375916 11787 heartbeater.cc:347] Connected to a master server at 127.25.123.62:41089
I20260928 15:07:06.376000 11787 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:06.376147 11787 heartbeater.cc:511] Master 127.25.123.62:41089 requested a full tablet report, sending...
I20260928 15:07:06.376574 11334 ts_manager.cc:194] Registered new tserver with Master: 84d5898785c140a497793fbec2725734 (127.25.123.3:36953)
I20260928 15:07:06.376947 11334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.3:49419
I20260928 15:07:06.382827 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:06.387303 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.384187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.25.123.62:41089)
0928 15:07:06.384283 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.384285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.384377 (+ 92us) server_negotiation.cc:407] Connection header received
0928 15:07:06.384482 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.384483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.384497 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.384554 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:06.384727 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.385243 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.386006 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.386203 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.386279 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.387027 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:06.387028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:06.387029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:06.387034 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:06.387054 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:06.387058 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:06.387120 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.387175 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.387202 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:07:06.388913 11334 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:06.390447 11334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:06.397361 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.392907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.25.123.2:34813)
0928 15:07:06.393223 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.393229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.393405 (+ 176us) server_negotiation.cc:407] Connection header received
0928 15:07:06.393449 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.393451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.393469 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.393569 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:06.393828 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.395063 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.396355 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.396914 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.397013 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.397119 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.397229 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.397267 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":95,"threads_started":1}
I20260928 15:07:06.397351 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.392802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:34813 (local address 127.0.0.1:59458)
0928 15:07:06.393207 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.393250 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.393323 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.393655 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.393658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.393660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:06.393734 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.393737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.395159 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.395160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.396285 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.396287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.397026 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.397027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.397117 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.397132 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.397207 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.397266 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":36,"thread_start_us":66,"threads_started":1}
I20260928 15:07:06.397541 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.392880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36953 (local address 127.0.0.1:49232)
0928 15:07:06.393298 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.393310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.393377 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.393708 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.393708 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.393710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:06.393771 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.393773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.394794 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.394795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.396217 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.396220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.397290 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.397291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.397358 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.397363 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.397415 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.397456 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":30,"thread_start_us":65,"threads_started":1}
I20260928 15:07:06.397644 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.393185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.25.123.1:37511)
0928 15:07:06.393396 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.393400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.393591 (+ 191us) server_negotiation.cc:407] Connection header received
0928 15:07:06.393659 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.393661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.393675 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.393768 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:06.393925 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.395084 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.396372 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.397096 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.397169 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.397251 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.397501 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.397545 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260928 15:07:06.397883 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.393007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.25.123.3:36953)
0928 15:07:06.393233 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.393239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.393366 (+ 127us) server_negotiation.cc:407] Connection header received
0928 15:07:06.393519 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.393522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.393539 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.393618 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:06.393850 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.394693 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.396312 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.397100 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.397169 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.397251 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.397745 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.397793 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260928 15:07:06.398187 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.393082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:37511 (local address 127.0.0.1:52262)
0928 15:07:06.393457 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.393487 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.393533 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.393828 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.393829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.393831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:06.393862 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.393863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.395175 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.395176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.396264 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.396267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.397291 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.397292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.397361 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.397363 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.397415 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.397457 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260928 15:07:06.398267 11589 tablet_service.cc:1467] Processing CreateTablet for tablet 0f83b1c07c6e4d998c91fdb53042c6d0 (DEFAULT_TABLE table=test-table [id=5fe531c928324945adc118c89b01fff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:06.398532 11721 tablet_service.cc:1467] Processing CreateTablet for tablet 0f83b1c07c6e4d998c91fdb53042c6d0 (DEFAULT_TABLE table=test-table [id=5fe531c928324945adc118c89b01fff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:06.398695 11589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f83b1c07c6e4d998c91fdb53042c6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:06.398876 11721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f83b1c07c6e4d998c91fdb53042c6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:06.401644 11801 tablet_bootstrap.cc:492] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Bootstrap starting.
I20260928 15:07:06.402348 11802 tablet_bootstrap.cc:492] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Bootstrap starting.
I20260928 15:07:06.402566 11801 tablet_bootstrap.cc:654] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:06.402720 11457 tablet_service.cc:1467] Processing CreateTablet for tablet 0f83b1c07c6e4d998c91fdb53042c6d0 (DEFAULT_TABLE table=test-table [id=5fe531c928324945adc118c89b01fff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:06.402945 11801 log.cc:824] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:06.403074 11802 tablet_bootstrap.cc:654] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:06.403141 11457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f83b1c07c6e4d998c91fdb53042c6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:06.403390 11802 log.cc:824] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:06.403841 11801 tablet_bootstrap.cc:492] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: No bootstrap required, opened a new log
I20260928 15:07:06.403913 11801 ts_tablet_manager.cc:1397] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:07:06.404146 11802 tablet_bootstrap.cc:492] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: No bootstrap required, opened a new log
I20260928 15:07:06.404234 11802 ts_tablet_manager.cc:1397] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:07:06.406221 11802 raft_consensus.cc:348] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.406412 11802 raft_consensus.cc:374] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:06.406461 11802 raft_consensus.cc:729] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d5898785c140a497793fbec2725734, State: Initialized, Role: FOLLOWER
I20260928 15:07:06.406592 11802 consensus_queue.cc:260] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.406586 11801 raft_consensus.cc:348] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.406747 11801 raft_consensus.cc:374] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:06.406785 11801 raft_consensus.cc:729] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccc7253e5a4219a15ccd5e937108c8, State: Initialized, Role: FOLLOWER
I20260928 15:07:06.406891 11802 ts_tablet_manager.cc:1428] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:06.406976 11805 tablet_bootstrap.cc:492] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83: Bootstrap starting.
I20260928 15:07:06.407019 11787 heartbeater.cc:503] Master 127.25.123.62:41089 was elected leader, sending a full tablet report...
I20260928 15:07:06.407193 11801 consensus_queue.cc:260] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.407478 11801 ts_tablet_manager.cc:1428] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 15:07:06.407612 11656 heartbeater.cc:503] Master 127.25.123.62:41089 was elected leader, sending a full tablet report...
I20260928 15:07:06.407641 11805 tablet_bootstrap.cc:654] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:06.407984 11805 log.cc:824] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:06.408906 11805 tablet_bootstrap.cc:492] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83: No bootstrap required, opened a new log
I20260928 15:07:06.408998 11805 ts_tablet_manager.cc:1397] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:07:06.411160 11805 raft_consensus.cc:348] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.411343 11805 raft_consensus.cc:374] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:06.411401 11805 raft_consensus.cc:729] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c067b0db200c4447a72b06750cc9ab83, State: Initialized, Role: FOLLOWER
I20260928 15:07:06.411540 11805 consensus_queue.cc:260] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.411815 11805 ts_tablet_manager.cc:1428] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:06.411899 11524 heartbeater.cc:503] Master 127.25.123.62:41089 was elected leader, sending a full tablet report...
W20260928 15:07:06.477792 11657 tablet.cc:2369] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:06.547452 11809 raft_consensus.cc:482] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:06.547587 11809 raft_consensus.cc:504] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.548007 11809 leader_election.cc:288] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccc7253e5a4219a15ccd5e937108c8 (127.25.123.2:34813), 84d5898785c140a497793fbec2725734 (127.25.123.3:36953)
I20260928 15:07:06.552318 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.548286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:54325 (local address 127.25.123.2:34813)
0928 15:07:06.548334 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.548336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.548347 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:06.548455 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.548457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.548471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.548533 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:06.548749 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.549562 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.551194 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.551886 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.551960 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.551967 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.552190 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.552241 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:07:06.552322 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.548427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:46467 (local address 127.25.123.3:36953)
0928 15:07:06.548498 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.548500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.548509 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:06.548622 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.548623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.548636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.548680 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:06.548844 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.549550 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.551194 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.551821 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.551955 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.551966 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.552190 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.552241 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 15:07:06.552340 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.548209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:34813 (local address 127.25.123.1:54325)
0928 15:07:06.548275 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.548289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.548337 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.548561 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.548562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.548564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:06.548645 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.548648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.549652 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.549652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.551068 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.551069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.551978 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.551979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.552057 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.552065 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.552146 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.552204 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":131}
I20260928 15:07:06.552434 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.548278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36953 (local address 127.25.123.1:46467)
0928 15:07:06.548444 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.548453 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:06.548500 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:06.548703 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:06.548704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:06.548705 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:06.548776 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.548778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.549651 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.549652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.551059 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:06.551063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.551958 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.551960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:06.552057 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.552065 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:07:06.552146 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:07:06.552201 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":57,"thread_start_us":87,"threads_started":1}
I20260928 15:07:06.552955 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83b1c07c6e4d998c91fdb53042c6d0" candidate_uuid: "c067b0db200c4447a72b06750cc9ab83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccc7253e5a4219a15ccd5e937108c8" is_pre_election: true
I20260928 15:07:06.553007 11741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83b1c07c6e4d998c91fdb53042c6d0" candidate_uuid: "c067b0db200c4447a72b06750cc9ab83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d5898785c140a497793fbec2725734" is_pre_election: true
I20260928 15:07:06.553177 11741 raft_consensus.cc:2454] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c067b0db200c4447a72b06750cc9ab83 in term 0.
I20260928 15:07:06.553177 11609 raft_consensus.cc:2454] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c067b0db200c4447a72b06750cc9ab83 in term 0.
I20260928 15:07:06.553519 11413 leader_election.cc:302] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccc7253e5a4219a15ccd5e937108c8, c067b0db200c4447a72b06750cc9ab83; no voters:
I20260928 15:07:06.553714 11809 raft_consensus.cc:2781] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:06.553790 11809 raft_consensus.cc:482] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:06.553831 11809 raft_consensus.cc:3037] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:06.554669 11809 raft_consensus.cc:504] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.554854 11809 leader_election.cc:288] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccc7253e5a4219a15ccd5e937108c8 (127.25.123.2:34813), 84d5898785c140a497793fbec2725734 (127.25.123.3:36953)
I20260928 15:07:06.555037 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83b1c07c6e4d998c91fdb53042c6d0" candidate_uuid: "c067b0db200c4447a72b06750cc9ab83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccc7253e5a4219a15ccd5e937108c8"
I20260928 15:07:06.555068 11741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83b1c07c6e4d998c91fdb53042c6d0" candidate_uuid: "c067b0db200c4447a72b06750cc9ab83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d5898785c140a497793fbec2725734"
I20260928 15:07:06.555124 11609 raft_consensus.cc:3037] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:06.555176 11741 raft_consensus.cc:3037] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:06.555900 11609 raft_consensus.cc:2454] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c067b0db200c4447a72b06750cc9ab83 in term 1.
I20260928 15:07:06.555962 11741 raft_consensus.cc:2454] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c067b0db200c4447a72b06750cc9ab83 in term 1.
I20260928 15:07:06.556147 11410 leader_election.cc:302] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d5898785c140a497793fbec2725734, c067b0db200c4447a72b06750cc9ab83; no voters:
I20260928 15:07:06.556250 11809 raft_consensus.cc:2781] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:06.556456 11809 raft_consensus.cc:686] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [term 1 LEADER]: Becoming Leader. State: Replica: c067b0db200c4447a72b06750cc9ab83, State: Running, Role: LEADER
I20260928 15:07:06.556584 11809 consensus_queue.cc:237] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } }
I20260928 15:07:06.557616 11334 catalog_manager.cc:5495] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 reported cstate change: term changed from 0 to 1, leader changed from <none> to c067b0db200c4447a72b06750cc9ab83 (127.25.123.1). New cstate: current_term: 1 leader_uuid: "c067b0db200c4447a72b06750cc9ab83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c067b0db200c4447a72b06750cc9ab83" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 37511 } health_report { overall_health: HEALTHY } } }
W20260928 15:07:06.580809 11525 tablet.cc:2369] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:06.587066 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.584254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.25.123.1:37511)
0928 15:07:06.584323 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:06.584326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:06.584363 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:07:06.584499 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:06.584500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:06.584531 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:06.584597 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:06.584828 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.585380 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.586172 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:06.586382 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:06.586454 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:06.586647 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:06.586781 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:06.586884 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:06.586908 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 15:07:06.586930 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:07:06.589357 11609 raft_consensus.cc:1261] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 1 FOLLOWER]: Refusing update from remote peer c067b0db200c4447a72b06750cc9ab83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:06.589371 11741 raft_consensus.cc:1261] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 1 FOLLOWER]: Refusing update from remote peer c067b0db200c4447a72b06750cc9ab83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:06.589775 11811 consensus_queue.cc:1035] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d5898785c140a497793fbec2725734" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:06.589916 11809 consensus_queue.cc:1035] T 0f83b1c07c6e4d998c91fdb53042c6d0 P c067b0db200c4447a72b06750cc9ab83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccc7253e5a4219a15ccd5e937108c8" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:06.592857 11814 mvcc.cc:205] Tried to move back new op lower bound from 7334330476906811392 to 7334330476776706048. Current Snapshot: MvccSnapshot[applied={T|T < 7334330476906811392}]
I20260928 15:07:06.592994 11815 mvcc.cc:205] Tried to move back new op lower bound from 7334330476906811392 to 7334330476776706048. Current Snapshot: MvccSnapshot[applied={T|T < 7334330476906811392}]
I20260928 15:07:06.593638 11817 mvcc.cc:205] Tried to move back new op lower bound from 7334330476906811392 to 7334330476776706048. Current Snapshot: MvccSnapshot[applied={T|T < 7334330476906811392}]
I20260928 15:07:06.595315 11334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:06.595431 11334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:06.596220 11334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92b5be458a2242678285c4d0d9998d46: Sending DeleteTablet for 3 replicas of tablet 0f83b1c07c6e4d998c91fdb53042c6d0
I20260928 15:07:06.596612 11589 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f83b1c07c6e4d998c91fdb53042c6d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:06 UTC) from {username='slave'} at 127.0.0.1:59458
I20260928 15:07:06.596630 11721 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f83b1c07c6e4d998c91fdb53042c6d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:06 UTC) from {username='slave'} at 127.0.0.1:49232
I20260928 15:07:06.597014 11825 tablet_replica.cc:330] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: stopping tablet replica
I20260928 15:07:06.597028 11457 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f83b1c07c6e4d998c91fdb53042c6d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:06 UTC) from {username='slave'} at 127.0.0.1:52262
I20260928 15:07:06.597131 11825 raft_consensus.cc:2229] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:06.597239 11825 raft_consensus.cc:2258] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:06.597254 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 11395
I20260928 15:07:06.598868 11826 tablet_replica.cc:330] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: stopping tablet replica
I20260928 15:07:06.598991 11826 raft_consensus.cc:2229] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:06.599098 11826 raft_consensus.cc:2258] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:06.599448 11825 ts_tablet_manager.cc:1905] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:06.599535 11826 ts_tablet_manager.cc:1905] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:06.601181 11825 ts_tablet_manager.cc:1918] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:06.601248 11825 log.cc:1196] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/0f83b1c07c6e4d998c91fdb53042c6d0
I20260928 15:07:06.601527 11826 ts_tablet_manager.cc:1918] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:06.601612 11826 log.cc:1196] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/0f83b1c07c6e4d998c91fdb53042c6d0
I20260928 15:07:06.601701 11825 ts_tablet_manager.cc:1939] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 52ccc7253e5a4219a15ccd5e937108c8: Deleting consensus metadata
I20260928 15:07:06.601867 11826 ts_tablet_manager.cc:1939] T 0f83b1c07c6e4d998c91fdb53042c6d0 P 84d5898785c140a497793fbec2725734: Deleting consensus metadata
I20260928 15:07:06.602474 11321 catalog_manager.cc:4841] TS 52ccc7253e5a4219a15ccd5e937108c8 (127.25.123.2:34813): tablet 0f83b1c07c6e4d998c91fdb53042c6d0 (table test-table [id=5fe531c928324945adc118c89b01fff8]) successfully deleted
I20260928 15:07:06.602643 11318 catalog_manager.cc:4841] TS 84d5898785c140a497793fbec2725734 (127.25.123.3:36953): tablet 0f83b1c07c6e4d998c91fdb53042c6d0 (table test-table [id=5fe531c928324945adc118c89b01fff8]) successfully deleted
W20260928 15:07:06.604425 11318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:37511 (error 108)
I20260928 15:07:06.604734 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 11527
I20260928 15:07:06.605394 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:06.604834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:37511 (local address 127.0.0.1:52274)
0928 15:07:06.605306 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:06.605353 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:37511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447}
W20260928 15:07:06.605803 11318 catalog_manager.cc:4584] TS c067b0db200c4447a72b06750cc9ab83 (127.25.123.1:37511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f83b1c07c6e4d998c91fdb53042c6d0: Network error: Client connection negotiation failed: client connection to 127.25.123.1:37511: connect: Connection refused (error 111)
I20260928 15:07:06.609951 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 11659
I20260928 15:07:06.614984 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 11303
2026-09-28T15:07:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (907 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:06 dist-test-slave-qnz0 krb5kdc[11831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:06 dist-test-slave-qnz0 krb5kdc[11831](info): set up 2 sockets
Sep 28 15:07:06 dist-test-slave-qnz0 krb5kdc[11831](info): commencing operation
krb5kdc: starting...
W20260928 15:07:07.692976 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:07 dist-test-slave-qnz0 krb5kdc[11831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:07.840117 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:34885
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:07.930502 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:07.930791 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:07.930835 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:07.932132 11844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:07.932201 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:07.932225 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:07.932246 11844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:07.932309 11844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:07.933956 11844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:07.934267 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:07.934522 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:07.939996 11849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:07.939934 11852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:07.939996 11850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:07.940707 11844 server_base.cc:1034] running on GCE node
I20260928 15:07:07.940966 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:07.941298 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:07.942484 11844 hybrid_clock.cc:648] HybridClock initialized: now 1790608027942473 us; error 35 us; skew 500 ppm
Sep 28 15:07:07 dist-test-slave-qnz0 krb5kdc[11831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608027, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:07.945613 11844 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:07.946271 11844 webserver.cc:460] Webserver started at http://127.25.123.62:45401/ using document root <none> and password file <none>
I20260928 15:07:07.946578 11844 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:07.946658 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:07.946835 11844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:07.948077 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "1584e58feb374bd48f21d2e59bb739b4"
format_stamp: "Formatted at 2026-09-28 15:07:07 on dist-test-slave-qnz0"
server_key: "b9ae9dd40a3fd84a04875b4322b17cfb"
server_key_iv: "0d4361fade3031250cc69d9633c20fed"
server_key_version: "encryptionkey@0"
I20260928 15:07:07.948537 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "1584e58feb374bd48f21d2e59bb739b4"
format_stamp: "Formatted at 2026-09-28 15:07:07 on dist-test-slave-qnz0"
server_key: "b9ae9dd40a3fd84a04875b4322b17cfb"
server_key_iv: "0d4361fade3031250cc69d9633c20fed"
server_key_version: "encryptionkey@0"
I20260928 15:07:07.950037 11844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:07.951006 11859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:07.951313 11844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:07.951505 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "1584e58feb374bd48f21d2e59bb739b4"
format_stamp: "Formatted at 2026-09-28 15:07:07 on dist-test-slave-qnz0"
server_key: "b9ae9dd40a3fd84a04875b4322b17cfb"
server_key_iv: "0d4361fade3031250cc69d9633c20fed"
server_key_version: "encryptionkey@0"
I20260928 15:07:07.951606 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:07.979779 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:07.982796 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:07.983017 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:07.987552 11844 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:34885
I20260928 15:07:07.987581 11912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:34885 every 8 connection(s)
I20260928 15:07:07.987979 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:07.988929 11913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:07.991200 11913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4: Bootstrap starting.
I20260928 15:07:07.991786 11913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:07.992048 11913 log.cc:824] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:07.992928 11913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4: No bootstrap required, opened a new log
I20260928 15:07:07.994580 11913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1584e58feb374bd48f21d2e59bb739b4" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34885 } }
I20260928 15:07:07.994730 11913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:07.994752 11913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1584e58feb374bd48f21d2e59bb739b4, State: Initialized, Role: FOLLOWER
I20260928 15:07:07.994868 11913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1584e58feb374bd48f21d2e59bb739b4" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34885 } }
I20260928 15:07:07.994974 11913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:07.995026 11913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:07.995100 11913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:07.995695 11913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1584e58feb374bd48f21d2e59bb739b4" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34885 } }
I20260928 15:07:07.995831 11913 leader_election.cc:302] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1584e58feb374bd48f21d2e59bb739b4; no voters:
I20260928 15:07:07.996037 11913 leader_election.cc:288] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:07.996133 11916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:07.996316 11916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1584e58feb374bd48f21d2e59bb739b4, State: Running, Role: LEADER
I20260928 15:07:07.996346 11913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:07.996464 11916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1584e58feb374bd48f21d2e59bb739b4" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34885 } }
I20260928 15:07:07.996599 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 11844
I20260928 15:07:07.996745 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:07.996946 26092 external_mini_cluster.cc:1428] Setting key 9384b7fe2015f2602ead7169089b56d1
I20260928 15:07:07.997149 11918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1584e58feb374bd48f21d2e59bb739b4. Latest consensus state: current_term: 1 leader_uuid: "1584e58feb374bd48f21d2e59bb739b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1584e58feb374bd48f21d2e59bb739b4" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34885 } } }
I20260928 15:07:07.997236 11918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:07.997687 11917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1584e58feb374bd48f21d2e59bb739b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1584e58feb374bd48f21d2e59bb739b4" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 34885 } } }
I20260928 15:07:07.997830 11917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:07.997859 11920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:07.998456 11920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:08.000547 11920 catalog_manager.cc:1347] Generated new cluster ID: 27605b4fe8bf4509a240ca0579f82b66
I20260928 15:07:08.000629 11920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:07:08 dist-test-slave-qnz0 krb5kdc[11831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:08.010272 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:07.997632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.25.123.62:34885)
0928 15:07:07.997862 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:07.997874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:07.997913 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:07:07.998067 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:07.998077 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:07.998145 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:07.998300 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:07.999344 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.000758 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.001858 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.002517 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.002641 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.005528 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:08.005534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:08.005535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:08.005551 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:08.008155 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.008660 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.008664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.008664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.008810 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.009041 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.009042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.009043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.009419 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:08.009609 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.009687 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.009992 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260928 15:07:08.022686 11920 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:08.023458 11920 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:08.029372 11920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4: Generated new TSK 0
I20260928 15:07:08.029587 11920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:07:08.059607 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34885
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:08.147321 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:08.147567 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:08.147593 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:08.148846 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:08.148926 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:08.148957 11939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:07:08.150496 11939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:08.150776 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:08.151033 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:08.156412 11947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:08.156483 11945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:08.156497 11944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:08.156622 11939 server_base.cc:1034] running on GCE node
I20260928 15:07:08.156898 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:08.157162 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:08.158345 11939 hybrid_clock.cc:648] HybridClock initialized: now 1790608028158338 us; error 40 us; skew 500 ppm
Sep 28 15:07:08 dist-test-slave-qnz0 krb5kdc[11831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:08.161062 11939 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:07:08.161528 11939 webserver.cc:460] Webserver started at http://127.25.123.1:34631/ using document root <none> and password file <none>
I20260928 15:07:08.161738 11939 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:08.161779 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:08.161876 11939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:08.162878 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "7e34e2911ddb9288221a8f22a1689452"
server_key_iv: "c0412ce9adcf8bfef2840300bc82c7a1"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.163167 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "7e34e2911ddb9288221a8f22a1689452"
server_key_iv: "c0412ce9adcf8bfef2840300bc82c7a1"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.164563 11939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 15:07:08.165402 11954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.165597 11939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:08.165683 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "7e34e2911ddb9288221a8f22a1689452"
server_key_iv: "c0412ce9adcf8bfef2840300bc82c7a1"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.165741 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:08.182374 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:08.184875 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:08.184995 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:08.185254 11939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:08.185586 11939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:08.185623 11939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.185647 11939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:08.185660 11939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.191738 11939 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:39105
I20260928 15:07:08.191820 12068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:39105 every 8 connection(s)
I20260928 15:07:08.192207 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:08.195464 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 11939
I20260928 15:07:08.195591 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:08.195791 26092 external_mini_cluster.cc:1428] Setting key 541ec8bb37f1b8a20830a5088b42be78
Sep 28 15:07:08 dist-test-slave-qnz0 krb5kdc[11831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:08.203836 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.193116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:58433 (local address 127.25.123.62:34885)
0928 15:07:08.193167 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.193169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.193417 (+ 248us) server_negotiation.cc:407] Connection header received
0928 15:07:08.193995 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.193997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.194012 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.194073 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:08.195049 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.195914 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.197143 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.197580 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.197655 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.200367 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:08.200369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:08.200370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:08.200378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:08.202104 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.202591 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.202593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.202593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.202656 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.202946 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.202947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.202948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.203083 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:08.203157 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.203481 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.203584 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:07:08.204066 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.193011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:34885 (local address 127.25.123.1:58433)
0928 15:07:08.193212 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.193256 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.193877 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.194128 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.194131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.194730 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:08.194943 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.194948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.196038 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.196039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.197040 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.197043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.197693 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.197694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.197826 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.198694 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:08.198704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:08.200253 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:08.202224 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:08.202235 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:08.202237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:08.202494 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:08.202729 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:08.202730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:08.202730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:08.202855 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:08.203159 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:08.203167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:08.203323 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.203454 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.203799 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260928 15:07:08.204651 12069 heartbeater.cc:347] Connected to a master server at 127.25.123.62:34885
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
I20260928 15:07:08.204766 12069 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:08.204957 12069 heartbeater.cc:511] Master 127.25.123.62:34885 requested a full tablet report, sending...
I20260928 15:07:08.205792 11876 ts_manager.cc:194] Registered new tserver with Master: 5a0bf1fbfb7a4de58e01acb8e08e3111 (127.25.123.1:39105)
I20260928 15:07:08.206388 11876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:58433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:07:08.225033 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34885
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:08.311084 12074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:08.311313 12074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:08.311340 12074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:08.312558 12074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:08.312605 12074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:08.312633 12074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:07:08.314136 12074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:08.314411 12074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:08.314687 12074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:08.320288 12082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:08.320330 12079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:08.320922 12074 server_base.cc:1034] running on GCE node
W20260928 15:07:08.321117 12080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:08.321408 12074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:08.321708 12074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:08.322903 12074 hybrid_clock.cc:648] HybridClock initialized: now 1790608028322859 us; error 84 us; skew 500 ppm
Sep 28 15:07:08 dist-test-slave-qnz0 krb5kdc[11831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:08.326717 12074 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:07:08.327199 12074 webserver.cc:460] Webserver started at http://127.25.123.2:41893/ using document root <none> and password file <none>
I20260928 15:07:08.327450 12074 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:08.327500 12074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:08.327627 12074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:08.328874 12074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "5809a2bf8533409f89e0959f7e03376c"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "2a10c83846731961198b408df4d36d4d"
server_key_iv: "35ca379cb2691658c8ef9efbde24df2a"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.329258 12074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "5809a2bf8533409f89e0959f7e03376c"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "2a10c83846731961198b408df4d36d4d"
server_key_iv: "35ca379cb2691658c8ef9efbde24df2a"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.330896 12074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:08.331876 12089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.332126 12074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:08.332209 12074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "5809a2bf8533409f89e0959f7e03376c"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "2a10c83846731961198b408df4d36d4d"
server_key_iv: "35ca379cb2691658c8ef9efbde24df2a"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.332324 12074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:08.348572 12074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:08.351111 12074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:08.351274 12074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:08.351541 12074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:08.351888 12074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:08.351944 12074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.351980 12074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:08.352007 12074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.358418 12074 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:37771
I20260928 15:07:08.358460 12203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:37771 every 8 connection(s)
I20260928 15:07:08.358803 12074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:08.360589 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 12074
I20260928 15:07:08.360693 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:08.360936 26092 external_mini_cluster.cc:1428] Setting key 003ae2126c59334b33a16aa7def94767
Sep 28 15:07:08 dist-test-slave-qnz0 krb5kdc[11831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:07:08.373493 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.359787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:57703 (local address 127.25.123.62:34885)
0928 15:07:08.359919 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.359923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.360255 (+ 332us) server_negotiation.cc:407] Connection header received
0928 15:07:08.361592 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.361594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.361608 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.361671 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:08.363383 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.364006 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.365434 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.365865 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.365990 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.369572 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:08.369576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:08.369576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:08.369590 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:08.371515 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.372098 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.372099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.372100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.372170 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.372625 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.372626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.372626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.372766 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:08.372822 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.373206 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.373288 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:07:08.373685 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.359607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:34885 (local address 127.25.123.2:57703)
0928 15:07:08.359949 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.360045 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.361446 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.361773 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.361779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.362753 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:08.363123 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.363136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.364130 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.364131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.365309 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.365313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.366064 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.366067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.366285 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.367134 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:08.367144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:08.369392 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:08.371647 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:08.371657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:08.371660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:08.371988 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:08.372295 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:08.372296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:08.372296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:08.372528 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:08.372847 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:08.372853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:08.373025 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.373129 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.373294 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260928 15:07:08.374258 12204 heartbeater.cc:347] Connected to a master server at 127.25.123.62:34885
I20260928 15:07:08.374418 12204 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:08.374688 12204 heartbeater.cc:511] Master 127.25.123.62:34885 requested a full tablet report, sending...
I20260928 15:07:08.375202 11876 ts_manager.cc:194] Registered new tserver with Master: 5809a2bf8533409f89e0959f7e03376c (127.25.123.2:37771)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
I20260928 15:07:08.375679 11876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:57703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:07:08.392488 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34885
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:08.478247 12209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:08.478466 12209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:08.478492 12209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:08.479674 12209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:08.479717 12209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:08.479743 12209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:07:08.481230 12209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:08.481501 12209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:08.481760 12209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:08.487232 12215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:08.487221 12217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:08.487215 12214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:08.487583 12209 server_base.cc:1034] running on GCE node
I20260928 15:07:08.487846 12209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:08.488070 12209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:08.489223 12209 hybrid_clock.cc:648] HybridClock initialized: now 1790608028489193 us; error 54 us; skew 500 ppm
Sep 28 15:07:08 dist-test-slave-qnz0 krb5kdc[11831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:08.491915 12209 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:07:08.492444 12209 webserver.cc:460] Webserver started at http://127.25.123.3:44301/ using document root <none> and password file <none>
I20260928 15:07:08.492714 12209 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:08.492784 12209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:08.492952 12209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:08.494040 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "2a59ffafaa7ac7be66a1802bc530e775"
server_key_iv: "2c66464ea57488f207320f8cf42da9b5"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.494406 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "2a59ffafaa7ac7be66a1802bc530e775"
server_key_iv: "2c66464ea57488f207320f8cf42da9b5"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.496188 12209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:07:08.497278 12224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.497540 12209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:08.497648 12209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7"
format_stamp: "Formatted at 2026-09-28 15:07:08 on dist-test-slave-qnz0"
server_key: "2a59ffafaa7ac7be66a1802bc530e775"
server_key_iv: "2c66464ea57488f207320f8cf42da9b5"
server_key_version: "encryptionkey@0"
I20260928 15:07:08.497745 12209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:08.509984 12209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:08.512645 12209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:08.512825 12209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:08.513087 12209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:08.513415 12209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:08.513471 12209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.513517 12209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:08.513548 12209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:08.520117 12209 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:33421
I20260928 15:07:08.520157 12337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:33421 every 8 connection(s)
I20260928 15:07:08.520535 12209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
Sep 28 15:07:08 dist-test-slave-qnz0 krb5kdc[11831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:08.527999 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 12209
I20260928 15:07:08.528149 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:08.528338 26092 external_mini_cluster.cc:1428] Setting key 0073d5858050ed944c8baa01ef1acd5f
I20260928 15:07:08.531476 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.521403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:48861 (local address 127.25.123.62:34885)
0928 15:07:08.521458 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.521461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.521641 (+ 180us) server_negotiation.cc:407] Connection header received
0928 15:07:08.522185 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.522186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.522197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.522259 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:08.523353 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.523893 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.525103 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.525297 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.525391 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.527995 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:08.527997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:08.527997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:08.528017 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:08.529861 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.530355 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.530356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.530356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.530430 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.530729 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.530730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.530731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.530866 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:08.530952 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.531273 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.531361 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 15:07:08.531716 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.521322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:34885 (local address 127.25.123.3:48861)
0928 15:07:08.521530 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.521568 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.522051 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.522404 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.522408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.523035 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:08.523249 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.523253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.524014 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.524015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.525015 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.525017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.525437 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.525438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.525570 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.526138 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:08.526147 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:08.527853 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:08.529987 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:08.529997 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:08.529999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:08.530257 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:08.530517 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:08.530518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:08.530518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:08.530646 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:08.530944 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:08.530951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:08.531111 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.531231 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.531433 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260928 15:07:08.532261 12338 heartbeater.cc:347] Connected to a master server at 127.25.123.62:34885
I20260928 15:07:08.532451 12338 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:08.532627 12338 heartbeater.cc:511] Master 127.25.123.62:34885 requested a full tablet report, sending...
I20260928 15:07:08.533033 11876 ts_manager.cc:194] Registered new tserver with Master: a9d25fda6dc44bec9fd8dce1dd11e8c7 (127.25.123.3:33421)
I20260928 15:07:08.533397 11876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:48861
I20260928 15:07:08.540305 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:08.548524 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.541716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.25.123.62:34885)
0928 15:07:08.541803 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.541805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.541902 (+ 97us) server_negotiation.cc:407] Connection header received
0928 15:07:08.541985 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.541987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.542005 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.542065 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:08.542677 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.543166 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.543919 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.544109 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.544197 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.545753 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:08.545755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:08.545755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:08.545764 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:08.547125 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.547587 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.547588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.547589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.547651 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:08.547892 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:08.547894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:08.547894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:08.548050 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:08.548174 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.548309 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.548376 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:07:08.550871 11876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:08.553388 11876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:08.561256 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.556338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:37771 (local address 127.0.0.1:42656)
0928 15:07:08.556624 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.556672 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.556757 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.557087 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.557090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.557093 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:08.557198 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.557200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.558705 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.558706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.559926 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.559928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.560891 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.560892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.560968 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.560997 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.561037 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.561173 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":76,"thread_start_us":148,"threads_started":1}
I20260928 15:07:08.561255 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.556823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.25.123.3:33421)
0928 15:07:08.557022 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.557026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.557169 (+ 143us) server_negotiation.cc:407] Connection header received
0928 15:07:08.557214 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.557217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.557234 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.557312 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:08.557540 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.558711 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.560059 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.560787 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.560865 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.560930 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.561107 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.561155 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260928 15:07:08.561353 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.556618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.25.123.2:37771)
0928 15:07:08.556820 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.556826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.556839 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:07:08.556889 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.556892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.556920 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.557078 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:08.557294 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.558586 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.560008 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.560771 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.561141 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.561201 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.561225 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.561270 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260928 15:07:08.561429 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.556624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.25.123.1:39105)
0928 15:07:08.557217 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.557221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.557232 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:08.557270 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.557273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.557287 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.557373 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:08.557491 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.558205 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.559834 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.560558 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.560842 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.560934 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.561168 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.561213 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":107,"threads_started":1}
I20260928 15:07:08.561679 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.556415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:33421 (local address 127.0.0.1:36994)
0928 15:07:08.557039 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.557050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.557110 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.557397 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.557398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.557400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:08.557452 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.557454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.558839 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.558840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.559981 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.559983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.560868 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.560869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.560975 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.560992 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.561079 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.561601 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":23,"thread_start_us":85,"threads_started":1}
I20260928 15:07:08.561842 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.556341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:39105 (local address 127.0.0.1:34740)
0928 15:07:08.556862 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.556875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.556928 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.557378 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.557379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.557382 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:08.557433 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.557435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.558320 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.558321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.559716 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.559720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.560965 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.560966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.561035 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.561038 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.561088 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.561784 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":613,"thread_start_us":195,"threads_started":1}
I20260928 15:07:08.562167 12137 tablet_service.cc:1467] Processing CreateTablet for tablet 404b24ebac8845f29fd8153213ec6311 (DEFAULT_TABLE table=test-table [id=745a91ace252466c883c54e176751e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:08.562566 12137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404b24ebac8845f29fd8153213ec6311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:08.562533 12272 tablet_service.cc:1467] Processing CreateTablet for tablet 404b24ebac8845f29fd8153213ec6311 (DEFAULT_TABLE table=test-table [id=745a91ace252466c883c54e176751e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:08.562573 12002 tablet_service.cc:1467] Processing CreateTablet for tablet 404b24ebac8845f29fd8153213ec6311 (DEFAULT_TABLE table=test-table [id=745a91ace252466c883c54e176751e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:08.562840 12272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404b24ebac8845f29fd8153213ec6311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:08.562952 12002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404b24ebac8845f29fd8153213ec6311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:08.565698 12355 tablet_bootstrap.cc:492] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111: Bootstrap starting.
I20260928 15:07:08.566274 12356 tablet_bootstrap.cc:492] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Bootstrap starting.
I20260928 15:07:08.566480 12357 tablet_bootstrap.cc:492] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Bootstrap starting.
I20260928 15:07:08.566534 12355 tablet_bootstrap.cc:654] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:08.567027 12355 log.cc:824] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:08.567148 12357 tablet_bootstrap.cc:654] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:08.567149 12356 tablet_bootstrap.cc:654] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:08.567829 12356 log.cc:824] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:08.567845 12357 log.cc:824] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:08.568683 12355 tablet_bootstrap.cc:492] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111: No bootstrap required, opened a new log
I20260928 15:07:08.568691 12357 tablet_bootstrap.cc:492] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: No bootstrap required, opened a new log
I20260928 15:07:08.568755 12357 ts_tablet_manager.cc:1397] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:07:08.568755 12355 ts_tablet_manager.cc:1397] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:08.569082 12356 tablet_bootstrap.cc:492] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: No bootstrap required, opened a new log
I20260928 15:07:08.569144 12356 ts_tablet_manager.cc:1397] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:08.570932 12356 raft_consensus.cc:348] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.570883 12355 raft_consensus.cc:348] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.570879 12357 raft_consensus.cc:348] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.571066 12356 raft_consensus.cc:374] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:08.571069 12357 raft_consensus.cc:374] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:08.571069 12355 raft_consensus.cc:374] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:08.571089 12356 raft_consensus.cc:729] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5809a2bf8533409f89e0959f7e03376c, State: Initialized, Role: FOLLOWER
I20260928 15:07:08.571106 12357 raft_consensus.cc:729] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d25fda6dc44bec9fd8dce1dd11e8c7, State: Initialized, Role: FOLLOWER
I20260928 15:07:08.571106 12355 raft_consensus.cc:729] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0bf1fbfb7a4de58e01acb8e08e3111, State: Initialized, Role: FOLLOWER
I20260928 15:07:08.571233 12357 consensus_queue.cc:260] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.571233 12355 consensus_queue.cc:260] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.571288 12356 consensus_queue.cc:260] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.571544 12357 ts_tablet_manager.cc:1428] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 15:07:08.571561 12355 ts_tablet_manager.cc:1428] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:08.571563 12338 heartbeater.cc:503] Master 127.25.123.62:34885 was elected leader, sending a full tablet report...
I20260928 15:07:08.571589 12356 ts_tablet_manager.cc:1428] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:07:08.571797 12204 heartbeater.cc:503] Master 127.25.123.62:34885 was elected leader, sending a full tablet report...
I20260928 15:07:08.571884 12069 heartbeater.cc:503] Master 127.25.123.62:34885 was elected leader, sending a full tablet report...
W20260928 15:07:08.609468 12205 tablet.cc:2369] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:07:08.693222 12070 tablet.cc:2369] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:07:08.771229 12339 tablet.cc:2369] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:08.782195 12362 raft_consensus.cc:482] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:08.782339 12362 raft_consensus.cc:504] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.782825 12362 leader_election.cc:288] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5809a2bf8533409f89e0959f7e03376c (127.25.123.2:37771), 5a0bf1fbfb7a4de58e01acb8e08e3111 (127.25.123.1:39105)
I20260928 15:07:08.786057 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.783299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:52183 (local address 127.25.123.1:39105)
0928 15:07:08.783371 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.783374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.783435 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:07:08.783490 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.783491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.783501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.783566 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:08.783756 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.784274 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.785300 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.785677 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.785763 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.785773 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.785933 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.785971 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:07:08.786075 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.783090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:52341 (local address 127.25.123.2:37771)
0928 15:07:08.783136 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.783138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.783149 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:08.783371 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.783372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.783391 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.783474 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:08.783829 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.784320 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.785316 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.785705 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.785798 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.785807 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.785962 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.786002 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:07:08.786082 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.783032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:37771 (local address 127.25.123.3:52341)
0928 15:07:08.783094 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.783107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.783157 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.783484 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.783486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.783488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:08.783698 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.783701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.784434 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.784435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.785204 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.785206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.785800 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.785801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.785873 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.785875 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.785952 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.786003 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":15}
I20260928 15:07:08.786094 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.783169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:39105 (local address 127.25.123.3:52183)
0928 15:07:08.783353 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.783365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:08.783423 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:08.783584 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:08.783585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:08.783587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:08.783686 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.783688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.784359 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.784360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.785205 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:08.785207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.785762 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.785763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:08.785836 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.785840 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:07:08.785924 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:07:08.785970 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":26,"thread_start_us":100,"threads_started":1}
I20260928 15:07:08.786494 12157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404b24ebac8845f29fd8153213ec6311" candidate_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5809a2bf8533409f89e0959f7e03376c" is_pre_election: true
I20260928 15:07:08.786664 12157 raft_consensus.cc:2454] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9d25fda6dc44bec9fd8dce1dd11e8c7 in term 0.
I20260928 15:07:08.786832 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404b24ebac8845f29fd8153213ec6311" candidate_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" is_pre_election: true
I20260928 15:07:08.786912 12227 leader_election.cc:302] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5809a2bf8533409f89e0959f7e03376c, a9d25fda6dc44bec9fd8dce1dd11e8c7; no voters:
I20260928 15:07:08.787021 12022 raft_consensus.cc:2454] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9d25fda6dc44bec9fd8dce1dd11e8c7 in term 0.
I20260928 15:07:08.787038 12362 raft_consensus.cc:2781] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:08.787113 12362 raft_consensus.cc:482] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:08.787148 12362 raft_consensus.cc:3037] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:08.787887 12362 raft_consensus.cc:504] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.788053 12362 leader_election.cc:288] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [CANDIDATE]: Term 1 election: Requested vote from peers 5809a2bf8533409f89e0959f7e03376c (127.25.123.2:37771), 5a0bf1fbfb7a4de58e01acb8e08e3111 (127.25.123.1:39105)
I20260928 15:07:08.788257 12157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404b24ebac8845f29fd8153213ec6311" candidate_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5809a2bf8533409f89e0959f7e03376c"
I20260928 15:07:08.788365 12157 raft_consensus.cc:3037] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:08.788270 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404b24ebac8845f29fd8153213ec6311" candidate_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111"
I20260928 15:07:08.788491 12022 raft_consensus.cc:3037] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:08.789235 12157 raft_consensus.cc:2454] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9d25fda6dc44bec9fd8dce1dd11e8c7 in term 1.
I20260928 15:07:08.789266 12022 raft_consensus.cc:2454] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9d25fda6dc44bec9fd8dce1dd11e8c7 in term 1.
I20260928 15:07:08.789443 12227 leader_election.cc:302] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5809a2bf8533409f89e0959f7e03376c, a9d25fda6dc44bec9fd8dce1dd11e8c7; no voters:
I20260928 15:07:08.789541 12362 raft_consensus.cc:2781] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:08.789708 12362 raft_consensus.cc:686] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 1 LEADER]: Becoming Leader. State: Replica: a9d25fda6dc44bec9fd8dce1dd11e8c7, State: Running, Role: LEADER
I20260928 15:07:08.789790 12362 consensus_queue.cc:237] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } }
I20260928 15:07:08.790738 11876 catalog_manager.cc:5495] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9d25fda6dc44bec9fd8dce1dd11e8c7 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9d25fda6dc44bec9fd8dce1dd11e8c7" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 } health_report { overall_health: UNKNOWN } } }
I20260928 15:07:08.851408 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.848724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.25.123.3:33421)
0928 15:07:08.848803 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:08.848806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:08.848817 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:08.848876 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:08.848877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:08.848899 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:08.848990 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:08.849154 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.849665 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.850511 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:08.850728 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:08.850813 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:08.850978 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:08.851085 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:08.851211 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:08.851269 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 15:07:08.851282 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:07:08.853672 12157 raft_consensus.cc:1261] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 1 FOLLOWER]: Refusing update from remote peer a9d25fda6dc44bec9fd8dce1dd11e8c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:08.853672 12022 raft_consensus.cc:1261] T 404b24ebac8845f29fd8153213ec6311 P 5a0bf1fbfb7a4de58e01acb8e08e3111 [term 1 FOLLOWER]: Refusing update from remote peer a9d25fda6dc44bec9fd8dce1dd11e8c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:08.854194 12365 consensus_queue.cc:1035] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0bf1fbfb7a4de58e01acb8e08e3111" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:08.854351 12362 consensus_queue.cc:1035] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5809a2bf8533409f89e0959f7e03376c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 37771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:08.856743 12369 mvcc.cc:205] Tried to move back new op lower bound from 7334330486181249024 to 7334330485923631104. Current Snapshot: MvccSnapshot[applied={T|T < 7334330486181249024}]
I20260928 15:07:08.860550 11876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:08.860689 11876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:08.861891 11876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1584e58feb374bd48f21d2e59bb739b4: Sending DeleteTablet for 3 replicas of tablet 404b24ebac8845f29fd8153213ec6311
I20260928 15:07:08.862337 12002 tablet_service.cc:1514] Processing DeleteTablet for tablet 404b24ebac8845f29fd8153213ec6311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:08 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:34740
I20260928 15:07:08.862471 12272 tablet_service.cc:1514] Processing DeleteTablet for tablet 404b24ebac8845f29fd8153213ec6311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:08 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:36994
I20260928 15:07:08.862697 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 11939
I20260928 15:07:08.862735 12137 tablet_service.cc:1514] Processing DeleteTablet for tablet 404b24ebac8845f29fd8153213ec6311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:08 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:42656
I20260928 15:07:08.862738 12379 tablet_replica.cc:330] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: stopping tablet replica
I20260928 15:07:08.863313 12379 raft_consensus.cc:2229] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:08.863454 12379 raft_consensus.cc:2258] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:08.863770 12380 tablet_replica.cc:330] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: stopping tablet replica
I20260928 15:07:08.863922 12380 raft_consensus.cc:2229] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:08.864020 12379 ts_tablet_manager.cc:1905] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:08.864118 12380 raft_consensus.cc:2258] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:08.864653 12380 ts_tablet_manager.cc:1905] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:08.866343 12379 ts_tablet_manager.cc:1918] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:08.866432 12379 log.cc:1196] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/404b24ebac8845f29fd8153213ec6311
I20260928 15:07:08.866720 12379 ts_tablet_manager.cc:1939] T 404b24ebac8845f29fd8153213ec6311 P a9d25fda6dc44bec9fd8dce1dd11e8c7: Deleting consensus metadata
I20260928 15:07:08.867035 12380 ts_tablet_manager.cc:1918] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:08.867174 12380 log.cc:1196] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/404b24ebac8845f29fd8153213ec6311
I20260928 15:07:08.867182 11861 catalog_manager.cc:4841] TS a9d25fda6dc44bec9fd8dce1dd11e8c7 (127.25.123.3:33421): tablet 404b24ebac8845f29fd8153213ec6311 (table test-table [id=745a91ace252466c883c54e176751e37]) successfully deleted
I20260928 15:07:08.867511 12380 ts_tablet_manager.cc:1939] T 404b24ebac8845f29fd8153213ec6311 P 5809a2bf8533409f89e0959f7e03376c: Deleting consensus metadata
I20260928 15:07:08.867967 11862 catalog_manager.cc:4841] TS 5809a2bf8533409f89e0959f7e03376c (127.25.123.2:37771): tablet 404b24ebac8845f29fd8153213ec6311 (table test-table [id=745a91ace252466c883c54e176751e37]) successfully deleted
W20260928 15:07:08.871281 11863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:39105 (error 108)
I20260928 15:07:08.871722 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 12074
I20260928 15:07:08.873827 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:08.873613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:39105 (local address 127.0.0.1:34750)
0928 15:07:08.873705 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:08.873771 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:39105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:07:08.874440 11863 catalog_manager.cc:4584] TS 5a0bf1fbfb7a4de58e01acb8e08e3111 (127.25.123.1:39105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 404b24ebac8845f29fd8153213ec6311: Network error: Client connection negotiation failed: client connection to 127.25.123.1:39105: connect: Connection refused (error 111)
I20260928 15:07:08.879138 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 12209
I20260928 15:07:08.884259 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 11844
2026-09-28T15:07:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2271 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T15:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:08Z Disabled control of system clock
I20260928 15:07:08.915369 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35627
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:09.010980 12385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:09.011250 12385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:09.011273 12385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:09.012697 12385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:09.012763 12385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:09.012777 12385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:09.012791 12385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:09.012804 12385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:09.014366 12385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:35627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:09.014652 12385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:09.015013 12385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:07:09.020664 12385 server_base.cc:1034] running on GCE node
W20260928 15:07:09.020610 12393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.020643 12391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.020642 12390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:09.021178 12385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:09.021572 12385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:09.022737 12385 hybrid_clock.cc:648] HybridClock initialized: now 1790608029022716 us; error 38 us; skew 500 ppm
I20260928 15:07:09.031103 12385 webserver.cc:460] Webserver started at http://127.25.123.62:37389/ using document root <none> and password file <none>
I20260928 15:07:09.031437 12385 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:09.031493 12385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:09.031648 12385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:09.032996 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "febed6c5b360780fd1e2b23b2c3bf3b4"
server_key_iv: "6d352fda708b5160486806adc4317c07"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.033414 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "febed6c5b360780fd1e2b23b2c3bf3b4"
server_key_iv: "6d352fda708b5160486806adc4317c07"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.034893 12385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:09.035887 12399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.036199 12385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:09.036342 12385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "febed6c5b360780fd1e2b23b2c3bf3b4"
server_key_iv: "6d352fda708b5160486806adc4317c07"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.036540 12385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:09.051712 12385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:09.052093 12385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:09.052260 12385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:09.056730 12385 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:35627
I20260928 15:07:09.056767 12451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:35627 every 8 connection(s)
I20260928 15:07:09.057153 12385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:09.058256 12452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:09.060931 12452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1: Bootstrap starting.
I20260928 15:07:09.061545 12452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:09.061843 12452 log.cc:824] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:09.061879 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 12385
I20260928 15:07:09.062014 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:09.062247 26092 external_mini_cluster.cc:1428] Setting key d494fcef994a5225fbc898110611d99e
I20260928 15:07:09.062798 12452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1: No bootstrap required, opened a new log
I20260928 15:07:09.064618 12452 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35627 } }
I20260928 15:07:09.064800 12452 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:09.064847 12452 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9796cc1aa7c4fe39a4a8ed01b9a6cc1, State: Initialized, Role: FOLLOWER
I20260928 15:07:09.065009 12452 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35627 } }
I20260928 15:07:09.065100 12452 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:09.065141 12452 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:09.065183 12452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:09.065773 12452 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35627 } }
I20260928 15:07:09.065898 12452 leader_election.cc:302] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9796cc1aa7c4fe39a4a8ed01b9a6cc1; no voters:
I20260928 15:07:09.066112 12452 leader_election.cc:288] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:09.066145 12457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:09.066249 12457 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [term 1 LEADER]: Becoming Leader. State: Replica: c9796cc1aa7c4fe39a4a8ed01b9a6cc1, State: Running, Role: LEADER
I20260928 15:07:09.066334 12457 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35627 } }
I20260928 15:07:09.066391 12452 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:09.066676 12458 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35627 } } }
I20260928 15:07:09.066777 12458 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:09.066679 12459 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9796cc1aa7c4fe39a4a8ed01b9a6cc1. Latest consensus state: current_term: 1 leader_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796cc1aa7c4fe39a4a8ed01b9a6cc1" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 35627 } } }
I20260928 15:07:09.067014 12459 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:09.067152 12462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:09.067713 12462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:09.069708 12462 catalog_manager.cc:1347] Generated new cluster ID: a62edff8083547dbba3f3e0c69554cae
I20260928 15:07:09.069777 12462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:09.070528 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.062867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.25.123.62:35627)
0928 15:07:09.063066 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.063073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.063099 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:07:09.064622 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.064631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.064708 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.064899 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:09.065182 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.066981 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.068269 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.068972 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.069107 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.069865 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:09.069869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:09.069870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:09.070054 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:09.070087 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:09.070099 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:09.070176 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.070264 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.070347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260928 15:07:09.080017 12462 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:09.080711 12462 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:09.086678 12462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1: Generated new TSK 0
I20260928 15:07:09.086933 12462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:09.090096 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35627
--builtin_ntp_servers=127.25.123.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:09.199146 12477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:09.199414 12477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:09.199443 12477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:09.201018 12477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:09.201078 12477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:09.201108 12477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:07:09.202623 12477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:09.202895 12477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:09.203296 12477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:09.208881 12482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.209053 12483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.208904 12485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:09.209153 12477 server_base.cc:1034] running on GCE node
I20260928 15:07:09.209770 12477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:09.210119 12477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:09.211360 12477 hybrid_clock.cc:648] HybridClock initialized: now 1790608029211343 us; error 34 us; skew 500 ppm
I20260928 15:07:09.212879 12477 webserver.cc:460] Webserver started at http://127.25.123.1:38677/ using document root <none> and password file <none>
I20260928 15:07:09.213207 12477 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:09.213282 12477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:09.213455 12477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:09.214596 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "c0cab9dfeafe41c9864a2a43a8188d2a"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "c61a5ff878120fd3541b56daf37c550f"
server_key_iv: "a67857d31b2d5b644e034d34ed5f0bdf"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.214991 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "c0cab9dfeafe41c9864a2a43a8188d2a"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "c61a5ff878120fd3541b56daf37c550f"
server_key_iv: "a67857d31b2d5b644e034d34ed5f0bdf"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.216598 12477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:09.217546 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.217792 12477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:09.217947 12477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "c0cab9dfeafe41c9864a2a43a8188d2a"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "c61a5ff878120fd3541b56daf37c550f"
server_key_iv: "a67857d31b2d5b644e034d34ed5f0bdf"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.218045 12477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:09.230929 12477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:09.231268 12477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:09.231483 12477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:09.231740 12477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:09.232095 12477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:09.232153 12477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.232200 12477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:09.232225 12477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.239017 12477 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:40193
I20260928 15:07:09.239117 12604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:40193 every 8 connection(s)
I20260928 15:07:09.239562 12477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:09.245172 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.240503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:44157 (local address 127.25.123.62:35627)
0928 15:07:09.240561 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.240563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.240837 (+ 274us) server_negotiation.cc:407] Connection header received
0928 15:07:09.241383 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.241384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.241396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.241458 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:09.242127 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.242642 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.243643 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.243832 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.243891 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.244888 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:09.244889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:09.244889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:09.244896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:09.244912 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:09.244916 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:09.244971 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.245078 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.245106 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:07:09.245416 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.240480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35627 (local address 127.25.123.1:44157)
0928 15:07:09.240688 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.240724 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.241244 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.241510 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.241513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.241521 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:09.242036 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.242041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.242742 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.242743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.243555 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.243556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.243934 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.243935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.244067 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.244713 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:09.244714 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:09.244744 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:09.244768 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:09.245002 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:09.245006 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:09.245008 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.245063 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.245153 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260928 15:07:09.246057 12605 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35627
I20260928 15:07:09.246176 12605 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:09.246366 12605 heartbeater.cc:511] Master 127.25.123.62:35627 requested a full tablet report, sending...
I20260928 15:07:09.246789 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 12477
I20260928 15:07:09.246924 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:09.246986 12416 ts_manager.cc:194] Registered new tserver with Master: c0cab9dfeafe41c9864a2a43a8188d2a (127.25.123.1:40193)
I20260928 15:07:09.247159 26092 external_mini_cluster.cc:1428] Setting key ec3075d2523825f97e317cf0d9567f25
I20260928 15:07:09.247793 12416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.1:44157
I20260928 15:07:09.248785 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35627
--builtin_ntp_servers=127.25.123.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:09.333643 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:09.333914 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:09.333952 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:09.335182 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:09.335263 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:09.335309 12608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:07:09.336815 12608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:09.337123 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:09.337461 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:07:09.343647 12608 server_base.cc:1034] running on GCE node
W20260928 15:07:09.343586 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.343583 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.343811 12617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:09.344027 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:09.344302 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:09.345502 12608 hybrid_clock.cc:648] HybridClock initialized: now 1790608029345480 us; error 39 us; skew 500 ppm
I20260928 15:07:09.346832 12608 webserver.cc:460] Webserver started at http://127.25.123.2:33603/ using document root <none> and password file <none>
I20260928 15:07:09.347105 12608 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:09.347178 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:09.347328 12608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:09.348497 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "5d52fa47afe445218dc7baf15a525c46"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "b3dc22aeac025c77b478f94ba345b0f6"
server_key_iv: "17192e55239877e54482f01830453ab8"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.348867 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "5d52fa47afe445218dc7baf15a525c46"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "b3dc22aeac025c77b478f94ba345b0f6"
server_key_iv: "17192e55239877e54482f01830453ab8"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.350250 12608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:07:09.351118 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.351367 12608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:09.351511 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "5d52fa47afe445218dc7baf15a525c46"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "b3dc22aeac025c77b478f94ba345b0f6"
server_key_iv: "17192e55239877e54482f01830453ab8"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.351594 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:09.370371 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:09.370811 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:09.371025 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:09.371310 12608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:09.371706 12608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:09.371773 12608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.371825 12608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:09.371878 12608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.378413 12608 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:44277
I20260928 15:07:09.378496 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:44277 every 8 connection(s)
I20260928 15:07:09.378863 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:09.384289 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 12608
I20260928 15:07:09.384522 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:09.384691 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.379727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:45773 (local address 127.25.123.62:35627)
0928 15:07:09.379777 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.379779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.380037 (+ 258us) server_negotiation.cc:407] Connection header received
0928 15:07:09.380571 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.380572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.380584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.380639 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:09.381317 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.381856 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.382929 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.383163 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.383236 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.384249 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:09.384251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:09.384251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:09.384258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:09.384276 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:09.384281 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:09.384415 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.384566 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.384604 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:07:09.384758 26092 external_mini_cluster.cc:1428] Setting key 99f608848628765d9e52d361896f9adc
I20260928 15:07:09.384960 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.379722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35627 (local address 127.25.123.2:45773)
0928 15:07:09.379909 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.379945 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.380475 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.380691 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.380695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.380703 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:09.381235 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.381240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.381958 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.381960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.382810 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.382813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.383270 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.383271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.383404 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.384082 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:09.384082 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:09.384111 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:09.384138 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:09.384419 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:09.384426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:09.384429 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.384560 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.384670 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 15:07:09.385485 12737 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35627
I20260928 15:07:09.385583 12737 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:09.385780 12737 heartbeater.cc:511] Master 127.25.123.62:35627 requested a full tablet report, sending...
I20260928 15:07:09.385846 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35627
--builtin_ntp_servers=127.25.123.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:09.386153 12415 ts_manager.cc:194] Registered new tserver with Master: 5d52fa47afe445218dc7baf15a525c46 (127.25.123.2:44277)
I20260928 15:07:09.386554 12415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.2:45773
W20260928 15:07:09.473845 12740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:09.474072 12740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:09.474094 12740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:09.475476 12740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:09.475539 12740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:09.475569 12740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:07:09.477177 12740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:09.477484 12740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:09.477828 12740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:09.483577 12748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.483577 12746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:09.483580 12745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:09.483912 12740 server_base.cc:1034] running on GCE node
I20260928 15:07:09.484136 12740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:09.484380 12740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:09.485600 12740 hybrid_clock.cc:648] HybridClock initialized: now 1790608029485586 us; error 31 us; skew 500 ppm
I20260928 15:07:09.487185 12740 webserver.cc:460] Webserver started at http://127.25.123.3:43297/ using document root <none> and password file <none>
I20260928 15:07:09.487502 12740 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:09.487561 12740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:09.487708 12740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:09.488986 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "a637ba2b94d44346a13146391fb75cfd"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "699d3445cfdc2dba883a79fc62e870ba"
server_key_iv: "65676c5fe146d4bdf11b869a26dd638f"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.489378 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "a637ba2b94d44346a13146391fb75cfd"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "699d3445cfdc2dba883a79fc62e870ba"
server_key_iv: "65676c5fe146d4bdf11b869a26dd638f"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.490814 12740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:07:09.491674 12755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.491899 12740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:09.492013 12740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "a637ba2b94d44346a13146391fb75cfd"
format_stamp: "Formatted at 2026-09-28 15:07:09 on dist-test-slave-qnz0"
server_key: "699d3445cfdc2dba883a79fc62e870ba"
server_key_iv: "65676c5fe146d4bdf11b869a26dd638f"
server_key_version: "encryptionkey@0"
I20260928 15:07:09.492097 12740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:09.512677 12740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:09.513039 12740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:09.513235 12740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:09.513522 12740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:09.513906 12740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:09.513967 12740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.514014 12740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:09.514062 12740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:09.520558 12740 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:36159
I20260928 15:07:09.520627 12868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:36159 every 8 connection(s)
I20260928 15:07:09.521034 12740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:09.521561 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 12740
I20260928 15:07:09.521669 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:09.521930 26092 external_mini_cluster.cc:1428] Setting key 43b71e6fe5f60790a21053d648c25a90
I20260928 15:07:09.526767 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.521970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:43417 (local address 127.25.123.62:35627)
0928 15:07:09.522030 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.522033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.522304 (+ 271us) server_negotiation.cc:407] Connection header received
0928 15:07:09.522955 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.522955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.522966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.523007 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:09.523729 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.524241 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.525291 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.525512 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.525584 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.526451 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:09.526452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:09.526452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:09.526457 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:09.526472 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:09.526476 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:09.526526 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.526669 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.526699 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:07:09.527083 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.521953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:35627 (local address 127.25.123.3:43417)
0928 15:07:09.522178 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.522225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.522829 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.523079 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.523082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.523090 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:09.523614 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.523618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.524351 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.524352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.525179 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.525181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.525612 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.525613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.525727 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.526280 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:09.526280 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:09.526309 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:09.526331 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:09.526572 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:09.526578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:09.526581 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.526663 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.526767 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
I20260928 15:07:09.527673 12869 heartbeater.cc:347] Connected to a master server at 127.25.123.62:35627
I20260928 15:07:09.527817 12869 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:09.528014 12869 heartbeater.cc:511] Master 127.25.123.62:35627 requested a full tablet report, sending...
I20260928 15:07:09.528519 12415 ts_manager.cc:194] Registered new tserver with Master: a637ba2b94d44346a13146391fb75cfd (127.25.123.3:36159)
I20260928 15:07:09.528964 12415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.3:43417
I20260928 15:07:09.534185 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:09.539268 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.535752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.25.123.62:35627)
0928 15:07:09.535805 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.535807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.535894 (+ 87us) server_negotiation.cc:407] Connection header received
0928 15:07:09.535939 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.535940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.535952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.536004 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:09.536181 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.536899 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.537774 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.537994 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.538064 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.538857 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:09.538858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:09.538858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:09.538863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:09.538877 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:09.538881 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:09.538978 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.539156 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.539171 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:07:09.541411 12415 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:09.542979 12415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:09.550266 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.545636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.25.123.1:40193)
0928 15:07:09.545847 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.545852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.545918 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:07:09.546005 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.546006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.546019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.546084 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:09.546293 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.547474 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.549132 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.549745 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.549848 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.549954 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.550097 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.550146 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260928 15:07:09.550283 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.545543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40193 (local address 127.0.0.1:42374)
0928 15:07:09.545850 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.545863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.545927 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.546117 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.546120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.546123 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:09.546210 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.546213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.547568 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.547569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.549023 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.549026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.549847 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.549848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.549942 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.549957 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.550074 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.550154 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":29,"thread_start_us":95,"threads_started":1}
I20260928 15:07:09.550282 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.545540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:44277 (local address 127.0.0.1:33032)
0928 15:07:09.545728 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.545772 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.545844 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.546768 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.546769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.546771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:09.546820 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.546822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.548089 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.548090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.549007 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.549010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.549971 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.549973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.550033 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.550037 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.550067 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.550149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260928 15:07:09.550488 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.545978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.25.123.3:36159)
0928 15:07:09.546195 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.546200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.546241 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:07:09.546432 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.546435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.546453 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.546528 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:09.546725 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.547868 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.549237 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.549940 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.549999 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.550088 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.550328 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.550368 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260928 15:07:09.550554 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.545761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:36159 (local address 127.0.0.1:54350)
0928 15:07:09.546178 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.546189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.546247 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.546599 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.546600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.546602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:09.546643 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.546645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.547964 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.547965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.549157 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.549159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.550121 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.550122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.550169 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.550173 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.550224 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.550263 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260928 15:07:09.551045 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.545678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.25.123.2:44277)
0928 15:07:09.546485 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.546489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.546498 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:09.546535 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.546537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.546551 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.546625 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:09.546918 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.547994 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.549103 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.549850 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.550772 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.550874 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.550909 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.550954 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":80,"threads_started":1}
I20260928 15:07:09.551397 12803 tablet_service.cc:1467] Processing CreateTablet for tablet d8694a5b7d7b4197b873ee33646adf10 (DEFAULT_TABLE table=test-table [id=52748ff01cdb460cbd7c854aa008ff52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:09.551409 12539 tablet_service.cc:1467] Processing CreateTablet for tablet d8694a5b7d7b4197b873ee33646adf10 (DEFAULT_TABLE table=test-table [id=52748ff01cdb460cbd7c854aa008ff52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:09.551662 12671 tablet_service.cc:1467] Processing CreateTablet for tablet d8694a5b7d7b4197b873ee33646adf10 (DEFAULT_TABLE table=test-table [id=52748ff01cdb460cbd7c854aa008ff52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:09.551815 12803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8694a5b7d7b4197b873ee33646adf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:09.551815 12539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8694a5b7d7b4197b873ee33646adf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:09.552001 12671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8694a5b7d7b4197b873ee33646adf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:09.555047 12883 tablet_bootstrap.cc:492] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Bootstrap starting.
I20260928 15:07:09.555540 12885 tablet_bootstrap.cc:492] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a: Bootstrap starting.
I20260928 15:07:09.555924 12883 tablet_bootstrap.cc:654] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:09.556186 12885 tablet_bootstrap.cc:654] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:09.556249 12883 log.cc:824] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:09.556535 12885 log.cc:824] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:09.556685 12884 tablet_bootstrap.cc:492] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Bootstrap starting.
I20260928 15:07:09.557163 12883 tablet_bootstrap.cc:492] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: No bootstrap required, opened a new log
I20260928 15:07:09.557246 12883 ts_tablet_manager.cc:1397] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:07:09.557407 12885 tablet_bootstrap.cc:492] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a: No bootstrap required, opened a new log
I20260928 15:07:09.557415 12884 tablet_bootstrap.cc:654] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:09.557485 12885 ts_tablet_manager.cc:1397] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:07:09.557756 12884 log.cc:824] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:09.558655 12884 tablet_bootstrap.cc:492] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: No bootstrap required, opened a new log
I20260928 15:07:09.558730 12884 ts_tablet_manager.cc:1397] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:07:09.559554 12885 raft_consensus.cc:348] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.559718 12885 raft_consensus.cc:374] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:09.559762 12885 raft_consensus.cc:729] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cab9dfeafe41c9864a2a43a8188d2a, State: Initialized, Role: FOLLOWER
I20260928 15:07:09.559883 12885 consensus_queue.cc:260] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.560240 12605 heartbeater.cc:503] Master 127.25.123.62:35627 was elected leader, sending a full tablet report...
I20260928 15:07:09.560364 12885 ts_tablet_manager.cc:1428] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:09.560779 12883 raft_consensus.cc:348] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.560957 12883 raft_consensus.cc:374] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:09.560997 12883 raft_consensus.cc:729] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d52fa47afe445218dc7baf15a525c46, State: Initialized, Role: FOLLOWER
I20260928 15:07:09.561120 12883 consensus_queue.cc:260] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.561702 12883 ts_tablet_manager.cc:1428] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:07:09.561863 12737 heartbeater.cc:503] Master 127.25.123.62:35627 was elected leader, sending a full tablet report...
I20260928 15:07:09.561802 12884 raft_consensus.cc:348] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.561971 12884 raft_consensus.cc:374] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:09.562011 12884 raft_consensus.cc:729] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a637ba2b94d44346a13146391fb75cfd, State: Initialized, Role: FOLLOWER
I20260928 15:07:09.562122 12884 consensus_queue.cc:260] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.562387 12884 ts_tablet_manager.cc:1428] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:07:09.562453 12869 heartbeater.cc:503] Master 127.25.123.62:35627 was elected leader, sending a full tablet report...
I20260928 15:07:09.606752 12891 raft_consensus.cc:482] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:09.606909 12891 raft_consensus.cc:504] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.607334 12891 leader_election.cc:288] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0cab9dfeafe41c9864a2a43a8188d2a (127.25.123.1:40193), 5d52fa47afe445218dc7baf15a525c46 (127.25.123.2:44277)
I20260928 15:07:09.611924 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.607509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40193 (local address 127.25.123.3:58487)
0928 15:07:09.607635 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.607647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.607698 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.607927 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.607928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.607930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:09.608007 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.608010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.609152 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.609153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.610617 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.610619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.611577 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.611578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.611673 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.611678 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.611752 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.611812 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":64}
I20260928 15:07:09.611956 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.607600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:52579 (local address 127.25.123.2:44277)
0928 15:07:09.607830 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.607832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.607841 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:09.608037 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.608039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.608054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.608113 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:09.608261 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.608999 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.610816 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.611391 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.611508 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.611521 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.611802 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.611858 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260928 15:07:09.611950 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.607660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:58487 (local address 127.25.123.1:40193)
0928 15:07:09.607709 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.607711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.607724 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:07:09.607758 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.607759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.607772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.607827 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:09.608080 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.609049 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.610793 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.611458 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.611537 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.611545 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.611797 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.611860 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:07:09.611940 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.607509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:44277 (local address 127.25.123.3:52579)
0928 15:07:09.607738 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.607750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:09.607814 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:09.608115 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:09.608116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:09.608117 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:09.608178 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.608179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.609137 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.609138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.610610 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:09.610614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.611602 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.611603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:09.611673 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.611678 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:07:09.611752 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:07:09.611809 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":45,"thread_start_us":105,"threads_started":1}
I20260928 15:07:09.612349 12559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8694a5b7d7b4197b873ee33646adf10" candidate_uuid: "a637ba2b94d44346a13146391fb75cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" is_pre_election: true
I20260928 15:07:09.612567 12559 raft_consensus.cc:2454] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a637ba2b94d44346a13146391fb75cfd in term 0.
I20260928 15:07:09.612887 12756 leader_election.cc:302] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a637ba2b94d44346a13146391fb75cfd, c0cab9dfeafe41c9864a2a43a8188d2a; no voters:
I20260928 15:07:09.612905 12691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8694a5b7d7b4197b873ee33646adf10" candidate_uuid: "a637ba2b94d44346a13146391fb75cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d52fa47afe445218dc7baf15a525c46" is_pre_election: true
I20260928 15:07:09.613041 12691 raft_consensus.cc:2454] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a637ba2b94d44346a13146391fb75cfd in term 0.
I20260928 15:07:09.613087 12891 raft_consensus.cc:2781] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:09.613137 12891 raft_consensus.cc:482] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:09.613174 12891 raft_consensus.cc:3037] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:09.613956 12891 raft_consensus.cc:504] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.614125 12891 leader_election.cc:288] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [CANDIDATE]: Term 1 election: Requested vote from peers c0cab9dfeafe41c9864a2a43a8188d2a (127.25.123.1:40193), 5d52fa47afe445218dc7baf15a525c46 (127.25.123.2:44277)
I20260928 15:07:09.614295 12691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8694a5b7d7b4197b873ee33646adf10" candidate_uuid: "a637ba2b94d44346a13146391fb75cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d52fa47afe445218dc7baf15a525c46"
I20260928 15:07:09.614319 12559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8694a5b7d7b4197b873ee33646adf10" candidate_uuid: "a637ba2b94d44346a13146391fb75cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a"
I20260928 15:07:09.614411 12559 raft_consensus.cc:3037] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:09.614418 12691 raft_consensus.cc:3037] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:09.615164 12559 raft_consensus.cc:2454] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a637ba2b94d44346a13146391fb75cfd in term 1.
I20260928 15:07:09.615272 12691 raft_consensus.cc:2454] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a637ba2b94d44346a13146391fb75cfd in term 1.
I20260928 15:07:09.615391 12756 leader_election.cc:302] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a637ba2b94d44346a13146391fb75cfd, c0cab9dfeafe41c9864a2a43a8188d2a; no voters:
I20260928 15:07:09.615592 12891 raft_consensus.cc:2781] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:09.615792 12891 raft_consensus.cc:686] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 1 LEADER]: Becoming Leader. State: Replica: a637ba2b94d44346a13146391fb75cfd, State: Running, Role: LEADER
I20260928 15:07:09.616012 12891 consensus_queue.cc:237] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } }
I20260928 15:07:09.617173 12415 catalog_manager.cc:5495] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd reported cstate change: term changed from 0 to 1, leader changed from <none> to a637ba2b94d44346a13146391fb75cfd (127.25.123.3). New cstate: current_term: 1 leader_uuid: "a637ba2b94d44346a13146391fb75cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a637ba2b94d44346a13146391fb75cfd" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 36159 } health_report { overall_health: HEALTHY } } }
I20260928 15:07:09.625180 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.622630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.25.123.3:36159)
0928 15:07:09.622694 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:09.622696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:09.622707 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:09.622746 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:09.622747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:09.622772 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:09.622858 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:09.623022 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.623504 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.624307 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:09.624585 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:09.624663 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:09.624805 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:09.624906 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:09.625022 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:09.625053 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 15:07:09.625062 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:07:09.627187 12559 raft_consensus.cc:1261] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 1 FOLLOWER]: Refusing update from remote peer a637ba2b94d44346a13146391fb75cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:09.627228 12691 raft_consensus.cc:1261] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 1 FOLLOWER]: Refusing update from remote peer a637ba2b94d44346a13146391fb75cfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:09.627612 12893 consensus_queue.cc:1035] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cab9dfeafe41c9864a2a43a8188d2a" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:09.627820 12891 consensus_queue.cc:1035] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d52fa47afe445218dc7baf15a525c46" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:07:09.629657 12738 tablet.cc:2369] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:09.630810 12897 mvcc.cc:205] Tried to move back new op lower bound from 7334330489350189056 to 7334330489308188672. Current Snapshot: MvccSnapshot[applied={T|T < 7334330489350189056}]
I20260928 15:07:09.631134 12898 mvcc.cc:205] Tried to move back new op lower bound from 7334330489350189056 to 7334330489308188672. Current Snapshot: MvccSnapshot[applied={T|T < 7334330489350189056}]
I20260928 15:07:09.633080 12416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:09.633204 12416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:09.634068 12416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9796cc1aa7c4fe39a4a8ed01b9a6cc1: Sending DeleteTablet for 3 replicas of tablet d8694a5b7d7b4197b873ee33646adf10
I20260928 15:07:09.634491 12803 tablet_service.cc:1514] Processing DeleteTablet for tablet d8694a5b7d7b4197b873ee33646adf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:09 UTC) from {username='slave'} at 127.0.0.1:54350
I20260928 15:07:09.634485 12539 tablet_service.cc:1514] Processing DeleteTablet for tablet d8694a5b7d7b4197b873ee33646adf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:09 UTC) from {username='slave'} at 127.0.0.1:42374
I20260928 15:07:09.634796 12908 tablet_replica.cc:330] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: stopping tablet replica
I20260928 15:07:09.634810 12671 tablet_service.cc:1514] Processing DeleteTablet for tablet d8694a5b7d7b4197b873ee33646adf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:09 UTC) from {username='slave'} at 127.0.0.1:33032
I20260928 15:07:09.634932 12908 raft_consensus.cc:2229] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:09.635051 12908 raft_consensus.cc:2258] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:09.635078 12909 tablet_replica.cc:330] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: stopping tablet replica
I20260928 15:07:09.635236 12907 tablet_replica.cc:330] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a: stopping tablet replica
I20260928 15:07:09.635342 12907 raft_consensus.cc:2229] T d8694a5b7d7b4197b873ee33646adf10 P c0cab9dfeafe41c9864a2a43a8188d2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:09.635380 12909 raft_consensus.cc:2229] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:09.635412 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 12477
I20260928 15:07:09.635548 12909 raft_consensus.cc:2258] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:09.636894 12909 ts_tablet_manager.cc:1905] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:09.639101 12909 ts_tablet_manager.cc:1918] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:09.639199 12909 log.cc:1196] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/d8694a5b7d7b4197b873ee33646adf10
I20260928 15:07:09.639483 12909 ts_tablet_manager.cc:1939] T d8694a5b7d7b4197b873ee33646adf10 P 5d52fa47afe445218dc7baf15a525c46: Deleting consensus metadata
I20260928 15:07:09.639765 12908 ts_tablet_manager.cc:1905] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:09.640030 12402 catalog_manager.cc:4841] TS 5d52fa47afe445218dc7baf15a525c46 (127.25.123.2:44277): tablet d8694a5b7d7b4197b873ee33646adf10 (table test-table [id=52748ff01cdb460cbd7c854aa008ff52]) successfully deleted
I20260928 15:07:09.641937 12908 ts_tablet_manager.cc:1918] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:09.642030 12908 log.cc:1196] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/d8694a5b7d7b4197b873ee33646adf10
I20260928 15:07:09.642297 12908 ts_tablet_manager.cc:1939] T d8694a5b7d7b4197b873ee33646adf10 P a637ba2b94d44346a13146391fb75cfd: Deleting consensus metadata
I20260928 15:07:09.642777 12400 catalog_manager.cc:4841] TS a637ba2b94d44346a13146391fb75cfd (127.25.123.3:36159): tablet d8694a5b7d7b4197b873ee33646adf10 (table test-table [id=52748ff01cdb460cbd7c854aa008ff52]) successfully deleted
W20260928 15:07:09.644820 12400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:40193 (error 108)
I20260928 15:07:09.645088 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 12608
I20260928 15:07:09.645442 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:09.645305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40193 (local address 127.0.0.1:42386)
0928 15:07:09.645362 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:09.645405 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:40193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 15:07:09.645819 12400 catalog_manager.cc:4584] TS c0cab9dfeafe41c9864a2a43a8188d2a (127.25.123.1:40193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8694a5b7d7b4197b873ee33646adf10: Network error: Client connection negotiation failed: client connection to 127.25.123.1:40193: connect: Connection refused (error 111)
I20260928 15:07:09.651041 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 12740
I20260928 15:07:09.655856 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 12385
2026-09-28T15:07:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (771 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:09 dist-test-slave-qnz0 krb5kdc[12913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:09 dist-test-slave-qnz0 krb5kdc[12913](info): set up 2 sockets
Sep 28 15:07:09 dist-test-slave-qnz0 krb5kdc[12913](info): commencing operation
krb5kdc: starting...
W20260928 15:07:10.719725 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:10 dist-test-slave-qnz0 krb5kdc[12913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:10.856717 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37019
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:10.944846 12926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:10.945178 12926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:10.945226 12926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:10.946620 12926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:10.946714 12926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:10.946740 12926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:10.946762 12926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:10.946791 12926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:10.948585 12926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:10.948913 12926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:10.949211 12926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:10.955087 12934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:10.955088 12931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:10.955206 12926 server_base.cc:1034] running on GCE node
W20260928 15:07:10.955087 12932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:10.955700 12926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:10.956094 12926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:10.957311 12926 hybrid_clock.cc:648] HybridClock initialized: now 1790608030957300 us; error 32 us; skew 500 ppm
Sep 28 15:07:10 dist-test-slave-qnz0 krb5kdc[12913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608030, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:10.960649 12926 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:10.961717 12926 webserver.cc:460] Webserver started at http://127.25.123.62:34401/ using document root <none> and password file <none>
I20260928 15:07:10.962085 12926 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:10.962139 12926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:10.962244 12926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:10.963400 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "460d8ca7e40142a09257bea288f4a681"
format_stamp: "Formatted at 2026-09-28 15:07:10 on dist-test-slave-qnz0"
server_key: "095d0b5e15192d2fd79fb9fd0c8a7610"
server_key_iv: "57720e21b4b284ba07a2125152b34c82"
server_key_version: "encryptionkey@0"
I20260928 15:07:10.963724 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "460d8ca7e40142a09257bea288f4a681"
format_stamp: "Formatted at 2026-09-28 15:07:10 on dist-test-slave-qnz0"
server_key: "095d0b5e15192d2fd79fb9fd0c8a7610"
server_key_iv: "57720e21b4b284ba07a2125152b34c82"
server_key_version: "encryptionkey@0"
I20260928 15:07:10.965330 12926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:07:10.966284 12941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:10.966524 12926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:10.966619 12926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "460d8ca7e40142a09257bea288f4a681"
format_stamp: "Formatted at 2026-09-28 15:07:10 on dist-test-slave-qnz0"
server_key: "095d0b5e15192d2fd79fb9fd0c8a7610"
server_key_iv: "57720e21b4b284ba07a2125152b34c82"
server_key_version: "encryptionkey@0"
I20260928 15:07:10.966682 12926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:10.977612 12926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:10.980078 12926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:10.980202 12926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:10.984659 12926 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:37019
I20260928 15:07:10.984742 12994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:37019 every 8 connection(s)
I20260928 15:07:10.985109 12926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:10.986297 12995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:10.988899 12995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681: Bootstrap starting.
I20260928 15:07:10.989562 12995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:10.989863 12995 log.cc:824] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:10.991067 12995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681: No bootstrap required, opened a new log
I20260928 15:07:10.992669 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 12926
I20260928 15:07:10.992822 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:10.993127 26092 external_mini_cluster.cc:1428] Setting key 237721743f330705fdb593d726a05c3a
I20260928 15:07:10.994024 12995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460d8ca7e40142a09257bea288f4a681" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37019 } }
I20260928 15:07:10.994184 12995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:10.994207 12995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460d8ca7e40142a09257bea288f4a681, State: Initialized, Role: FOLLOWER
I20260928 15:07:10.994349 12995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460d8ca7e40142a09257bea288f4a681" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37019 } }
I20260928 15:07:10.994427 12995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:10.994449 12995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:10.994518 12995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:10.995764 12995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460d8ca7e40142a09257bea288f4a681" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37019 } }
I20260928 15:07:10.995869 12995 leader_election.cc:302] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460d8ca7e40142a09257bea288f4a681; no voters:
I20260928 15:07:10.996035 12995 leader_election.cc:288] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:10.996063 13000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:10.996193 13000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [term 1 LEADER]: Becoming Leader. State: Replica: 460d8ca7e40142a09257bea288f4a681, State: Running, Role: LEADER
I20260928 15:07:10.996281 13000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460d8ca7e40142a09257bea288f4a681" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37019 } }
I20260928 15:07:10.996335 12995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:10.996644 13000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "460d8ca7e40142a09257bea288f4a681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460d8ca7e40142a09257bea288f4a681" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37019 } } }
I20260928 15:07:10.996749 13000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:10.996891 13000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460d8ca7e40142a09257bea288f4a681. Latest consensus state: current_term: 1 leader_uuid: "460d8ca7e40142a09257bea288f4a681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460d8ca7e40142a09257bea288f4a681" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37019 } } }
I20260928 15:07:10.996963 13000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:10.997257 13006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:10.997809 13006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:10.999547 13006 catalog_manager.cc:1347] Generated new cluster ID: 7c02b8c3dd4f4f52b0aca8eb657172ef
I20260928 15:07:10.999608 13006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:07:10 dist-test-slave-qnz0 krb5kdc[12913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:11.007149 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:10.993819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.25.123.62:37019)
0928 15:07:10.994063 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:10.994072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:10.994101 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:07:10.994235 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:10.994244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:10.994337 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:10.994521 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:10.995461 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:10.997407 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:10.998563 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:10.999240 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:10.999375 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.002885 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:11.002891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:11.002891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:11.002906 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:11.005217 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.005721 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.005724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.005725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.005876 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.006108 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.006108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.006108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.006433 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:11.006592 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.006734 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.006893 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":123,"threads_started":1}
I20260928 15:07:11.016991 13006 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:11.017613 13006 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:11.022282 13006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681: Generated new TSK 0
I20260928 15:07:11.022488 13006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.123.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
Entry for principal kudu/127.25.123.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab.
I20260928 15:07:11.056654 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37019
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:11.148805 13021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:11.149047 13021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:11.149076 13021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:11.150504 13021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:11.150571 13021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:11.150725 13021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:07:11.152882 13021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.1.keytab
--principal=kudu/127.25.123.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:11.153146 13021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:11.153348 13021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:11.158711 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.158712 13027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.158715 13026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:11.159157 13021 server_base.cc:1034] running on GCE node
I20260928 15:07:11.159492 13021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:11.159754 13021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:11.160931 13021 hybrid_clock.cc:648] HybridClock initialized: now 1790608031160914 us; error 43 us; skew 500 ppm
Sep 28 15:07:11 dist-test-slave-qnz0 krb5kdc[12913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608031, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:11.163748 13021 init.cc:377] Logged in from keytab as kudu/127.25.123.1@KRBTEST.COM (short username kudu)
I20260928 15:07:11.164232 13021 webserver.cc:460] Webserver started at http://127.25.123.1:36829/ using document root <none> and password file <none>
I20260928 15:07:11.164584 13021 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:11.164659 13021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:11.164820 13021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:11.165954 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "fb5a3764bb6246faac9ef4a992d3f999"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "9371f31c1ba6b49c6f7aca9c09437e91"
server_key_iv: "cdde83186dbd5a3d3dd3870b7eb89daf"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.166422 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "fb5a3764bb6246faac9ef4a992d3f999"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "9371f31c1ba6b49c6f7aca9c09437e91"
server_key_iv: "cdde83186dbd5a3d3dd3870b7eb89daf"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.168071 13021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:11.169094 13036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.169279 13021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:07:11.169387 13021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "fb5a3764bb6246faac9ef4a992d3f999"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "9371f31c1ba6b49c6f7aca9c09437e91"
server_key_iv: "cdde83186dbd5a3d3dd3870b7eb89daf"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.169544 13021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:11.203586 13021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:11.206369 13021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:11.206540 13021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:11.206779 13021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:11.207111 13021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:11.207166 13021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.207211 13021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:11.207238 13021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.214972 13021 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:38221
I20260928 15:07:11.215049 13150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:38221 every 8 connection(s)
I20260928 15:07:11.215444 13021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
Sep 28 15:07:11 dist-test-slave-qnz0 krb5kdc[12913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608031, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.1@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:11.223037 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 13021
I20260928 15:07:11.223161 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:11.223419 26092 external_mini_cluster.cc:1428] Setting key b95bd936318c9eb64550e0b6236954bb
I20260928 15:07:11.226724 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.216307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:44253 (local address 127.25.123.62:37019)
0928 15:07:11.216379 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.216381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.216658 (+ 277us) server_negotiation.cc:407] Connection header received
0928 15:07:11.217409 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.217411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.217439 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.217522 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:11.218817 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.219493 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.220572 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.220766 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.220822 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.223090 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:11.223092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:11.223092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:11.223099 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:11.224993 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.225530 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.225531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.225532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.225600 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.225937 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.225938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.225939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.226094 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:11.226170 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.226485 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.226598 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 15:07:11.226768 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.216272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37019 (local address 127.25.123.1:44253)
0928 15:07:11.216466 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.216519 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.217255 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.217609 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.217615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.218462 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:11.218723 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.218728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.219609 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.219610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.220501 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.220504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.220840 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.220840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.220942 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.221447 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:11.221456 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:11.222975 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:11.225080 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:11.225090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:11.225092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:11.225437 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:11.225692 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:11.225693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:11.225694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:11.225855 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:11.226191 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:11.226198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:11.226333 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.226442 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.226591 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260928 15:07:11.227348 13151 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37019
I20260928 15:07:11.227454 13151 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:11.227660 13151 heartbeater.cc:511] Master 127.25.123.62:37019 requested a full tablet report, sending...
I20260928 15:07:11.228195 12958 ts_manager.cc:194] Registered new tserver with Master: fb5a3764bb6246faac9ef4a992d3f999 (127.25.123.1:38221)
I20260928 15:07:11.228876 12958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.1@KRBTEST.COM'} at 127.25.123.1:44253
WARNING: no policy specified for kudu/127.25.123.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
Entry for principal kudu/127.25.123.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab.
I20260928 15:07:11.252765 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37019
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:11.339164 13156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:11.339404 13156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:11.339432 13156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:11.340694 13156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:11.340744 13156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:11.340772 13156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:07:11.342347 13156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.2.keytab
--principal=kudu/127.25.123.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:11.342639 13156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:11.342932 13156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:11.348546 13161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:11.348675 13156 server_base.cc:1034] running on GCE node
W20260928 15:07:11.348567 13162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.348599 13164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:11.349097 13156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:11.349316 13156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:11.350484 13156 hybrid_clock.cc:648] HybridClock initialized: now 1790608031350468 us; error 38 us; skew 500 ppm
Sep 28 15:07:11 dist-test-slave-qnz0 krb5kdc[12913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608031, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:11.353546 13156 init.cc:377] Logged in from keytab as kudu/127.25.123.2@KRBTEST.COM (short username kudu)
I20260928 15:07:11.354010 13156 webserver.cc:460] Webserver started at http://127.25.123.2:41597/ using document root <none> and password file <none>
I20260928 15:07:11.354243 13156 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:11.354288 13156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:11.354388 13156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:11.355631 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "dd937607981343478892fe119f9ad1d1"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "9e109ae32e395664099309539792210b"
server_key_iv: "865f17fc89aba56830b0413f02104a53"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.355960 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "dd937607981343478892fe119f9ad1d1"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "9e109ae32e395664099309539792210b"
server_key_iv: "865f17fc89aba56830b0413f02104a53"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.357672 13156 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:07:11.358722 13171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.359032 13156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:11.359151 13156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "dd937607981343478892fe119f9ad1d1"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "9e109ae32e395664099309539792210b"
server_key_iv: "865f17fc89aba56830b0413f02104a53"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.359241 13156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:11.375458 13156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:11.377995 13156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:11.378226 13156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:11.378491 13156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:11.379000 13156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:11.379066 13156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.379113 13156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:11.379138 13156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.385099 13156 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:34477
I20260928 15:07:11.385147 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:34477 every 8 connection(s)
I20260928 15:07:11.385703 13156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:11.388608 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 13156
I20260928 15:07:11.388737 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:11.388938 26092 external_mini_cluster.cc:1428] Setting key b43ab0c904137c4e23b92379bdb80b21
Sep 28 15:07:11 dist-test-slave-qnz0 krb5kdc[12913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608031, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.2@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:11.397565 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.386619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:51511 (local address 127.25.123.62:37019)
0928 15:07:11.386700 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.386702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.386930 (+ 228us) server_negotiation.cc:407] Connection header received
0928 15:07:11.387511 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.387512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.387523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.387588 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:11.388711 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.389468 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.390685 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.390924 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.391030 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.393979 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:11.393980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:11.393981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:11.393989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:11.395547 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.396266 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.396267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.396267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.396327 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.396695 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.396696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.396696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.396862 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:11.396923 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.397396 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.397488 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:07:11.397928 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.386571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37019 (local address 127.25.123.2:51511)
0928 15:07:11.386776 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.386830 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.387370 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.387670 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.387674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.388349 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:11.388590 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.388595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.389572 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.389573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.390577 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.390580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.391042 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.391043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.391191 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.391818 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:11.391828 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:11.393842 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:11.395653 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:11.395664 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:11.395666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:11.396107 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:11.396431 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:11.396432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:11.396433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:11.396593 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:11.396947 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:11.396955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:11.397183 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.397345 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.397569 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.25.123.3@KRBTEST.COM; defaulting to no policy
I20260928 15:07:11.398485 13286 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37019
I20260928 15:07:11.398603 13286 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:11.398792 13286 heartbeater.cc:511] Master 127.25.123.62:37019 requested a full tablet report, sending...
I20260928 15:07:11.399212 12958 ts_manager.cc:194] Registered new tserver with Master: dd937607981343478892fe119f9ad1d1 (127.25.123.2:34477)
I20260928 15:07:11.399646 12958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.2@KRBTEST.COM'} at 127.25.123.2:51511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
Entry for principal kudu/127.25.123.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab.
I20260928 15:07:11.422188 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37019
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:11.517288 13291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:11.517560 13291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:11.517587 13291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:11.518999 13291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:11.519083 13291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:11.519114 13291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:07:11.520733 13291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.3.keytab
--principal=kudu/127.25.123.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:11.521008 13291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:11.521267 13291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:07:11.527257 13291 server_base.cc:1034] running on GCE node
W20260928 15:07:11.527253 13296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.527253 13297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.527274 13299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:11.527855 13291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:11.528136 13291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:11.529307 13291 hybrid_clock.cc:648] HybridClock initialized: now 1790608031529281 us; error 39 us; skew 500 ppm
Sep 28 15:07:11 dist-test-slave-qnz0 krb5kdc[12913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608031, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:11.532275 13291 init.cc:377] Logged in from keytab as kudu/127.25.123.3@KRBTEST.COM (short username kudu)
I20260928 15:07:11.532758 13291 webserver.cc:460] Webserver started at http://127.25.123.3:38689/ using document root <none> and password file <none>
I20260928 15:07:11.532996 13291 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:11.533090 13291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:11.533239 13291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:11.534325 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "cddab053d8dd446fa2623f0f941931c0"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "1e03f7b4ffb77d1c702b8f694bbd8a98"
server_key_iv: "115a28a9e26025f73d1fb2a21c6fd919"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.534695 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "cddab053d8dd446fa2623f0f941931c0"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "1e03f7b4ffb77d1c702b8f694bbd8a98"
server_key_iv: "115a28a9e26025f73d1fb2a21c6fd919"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.536263 13291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:07:11.537261 13306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.537626 13291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:11.537873 13291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "cddab053d8dd446fa2623f0f941931c0"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "1e03f7b4ffb77d1c702b8f694bbd8a98"
server_key_iv: "115a28a9e26025f73d1fb2a21c6fd919"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.537962 13291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:11.552649 13291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:11.555749 13291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:11.556010 13291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:11.556355 13291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:11.556905 13291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:11.556970 13291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.557009 13291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:11.557040 13291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.564101 13291 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:35337
I20260928 15:07:11.564141 13420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:35337 every 8 connection(s)
I20260928 15:07:11.564540 13291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:11.568953 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 13291
I20260928 15:07:11.569070 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:11.569269 26092 external_mini_cluster.cc:1428] Setting key 3429dd9ed59d57365a01a5436197a0b2
Sep 28 15:07:11 dist-test-slave-qnz0 krb5kdc[12913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608031, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.3@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:11.576737 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.565397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:47089 (local address 127.25.123.62:37019)
0928 15:07:11.565525 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.565528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.565684 (+ 156us) server_negotiation.cc:407] Connection header received
0928 15:07:11.566244 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.566245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.566259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.566311 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:11.567335 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.567882 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.569214 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.569446 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.569563 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.573355 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:11.573357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:11.573358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:11.573367 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:11.574923 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.575487 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.575488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.575489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.575566 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.575881 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.575882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.575882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.576021 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:11.576085 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.576583 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.576662 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:07:11.577057 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.565295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37019 (local address 127.25.123.3:47089)
0928 15:07:11.565558 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.565603 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.566128 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.566390 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.566394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.567007 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:11.567226 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.567231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.567985 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.567986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.569065 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.569069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.569635 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.569636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.569814 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.570764 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:11.570781 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:11.573197 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:11.575067 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:11.575078 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:11.575080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:11.575376 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:11.575666 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:11.575667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:11.575667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:11.575791 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:11.576123 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:11.576129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:11.576356 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.576552 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.576728 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":113,"threads_started":1}
I20260928 15:07:11.577749 13421 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37019
I20260928 15:07:11.577883 13421 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:11.578102 13421 heartbeater.cc:511] Master 127.25.123.62:37019 requested a full tablet report, sending...
I20260928 15:07:11.578646 12958 ts_manager.cc:194] Registered new tserver with Master: cddab053d8dd446fa2623f0f941931c0 (127.25.123.3:35337)
I20260928 15:07:11.579087 12958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.123.3@KRBTEST.COM'} at 127.25.123.3:47089
I20260928 15:07:11.582146 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:11.590085 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.583685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.25.123.62:37019)
0928 15:07:11.583750 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.583753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.583763 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:07:11.583822 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.583823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.583833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.583902 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:11.584528 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.585034 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.585773 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.585970 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.586041 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.587321 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:11.587323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:11.587323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:11.587331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:11.588847 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.589205 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.589206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.589207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.589262 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:11.589459 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:11.589460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:11.589460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:11.589634 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:11.589744 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.589868 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.589954 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:07:11.591866 12958 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:11.593624 12958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:11.601459 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.596298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:34477 (local address 127.0.0.1:42246)
0928 15:07:11.596591 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.596602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.596671 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.597103 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.597107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.597110 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:11.597204 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.597207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.598800 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.598801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.600158 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.600162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.601186 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.601187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.601273 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.601281 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.601315 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.601363 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":42,"thread_start_us":203,"threads_started":1}
I20260928 15:07:11.601466 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.596442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.25.123.1:38221)
0928 15:07:11.596895 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.596899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.596908 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:11.596941 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.596942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.596955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.597221 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:11.597795 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.598557 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.600272 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.601075 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.601157 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.601217 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.601322 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.601361 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":107,"threads_started":1}
I20260928 15:07:11.601837 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.596695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.25.123.2:34477)
0928 15:07:11.596902 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.596907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.596920 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:07:11.596964 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.596967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.596985 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.597297 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:11.597380 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.598686 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.600272 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.601075 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.601604 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.601677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.601705 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.601750 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260928 15:07:11.601877 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.596295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:38221 (local address 127.0.0.1:47834)
0928 15:07:11.596556 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.596598 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.596669 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.597261 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.597262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.597265 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:11.597312 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.597315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.598668 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.598669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.600159 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.600162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.601152 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.601153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.601225 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.601230 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.601295 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.601330 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":74,"thread_start_us":70,"threads_started":1}
I20260928 15:07:11.602005 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.596298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:35337 (local address 127.0.0.1:38602)
0928 15:07:11.596890 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.596902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.596962 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.597669 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.597670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.597673 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:11.597721 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.597722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.599175 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.599176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.600114 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.600118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.601022 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.601023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.601131 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.601151 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.601231 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.601308 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260928 15:07:11.601859 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.596654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.25.123.3:35337)
0928 15:07:11.597487 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.597491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.597501 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:07:11.597539 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.597541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.597559 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.597650 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:11.597828 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.598950 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.600253 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.600914 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.601019 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.601091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.601421 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.601469 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":81,"threads_started":1}
I20260928 15:07:11.602667 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 9effb7c5ed404a88ba94dac0d58deb33 (DEFAULT_TABLE table=test-table [id=69b284d9e0f3471090368b7e8d5540a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:11.602655 13220 tablet_service.cc:1467] Processing CreateTablet for tablet 9effb7c5ed404a88ba94dac0d58deb33 (DEFAULT_TABLE table=test-table [id=69b284d9e0f3471090368b7e8d5540a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:11.602793 13354 tablet_service.cc:1467] Processing CreateTablet for tablet 9effb7c5ed404a88ba94dac0d58deb33 (DEFAULT_TABLE table=test-table [id=69b284d9e0f3471090368b7e8d5540a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:11.602985 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9effb7c5ed404a88ba94dac0d58deb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:11.603014 13220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9effb7c5ed404a88ba94dac0d58deb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:11.603084 13354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9effb7c5ed404a88ba94dac0d58deb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:11.606079 13438 tablet_bootstrap.cc:492] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999: Bootstrap starting.
I20260928 15:07:11.606501 13439 tablet_bootstrap.cc:492] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Bootstrap starting.
I20260928 15:07:11.607076 13438 tablet_bootstrap.cc:654] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:11.607183 13439 tablet_bootstrap.cc:654] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:11.607787 13438 log.cc:824] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:11.607952 13439 log.cc:824] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:11.608342 13440 tablet_bootstrap.cc:492] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Bootstrap starting.
I20260928 15:07:11.608814 13438 tablet_bootstrap.cc:492] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999: No bootstrap required, opened a new log
I20260928 15:07:11.608906 13438 ts_tablet_manager.cc:1397] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:11.609023 13440 tablet_bootstrap.cc:654] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:11.609328 13439 tablet_bootstrap.cc:492] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: No bootstrap required, opened a new log
I20260928 15:07:11.609417 13439 ts_tablet_manager.cc:1397] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:07:11.609678 13440 log.cc:824] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:11.610543 13440 tablet_bootstrap.cc:492] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: No bootstrap required, opened a new log
I20260928 15:07:11.610621 13440 ts_tablet_manager.cc:1397] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:07:11.610973 13438 raft_consensus.cc:348] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.611147 13438 raft_consensus.cc:374] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:11.611191 13438 raft_consensus.cc:729] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5a3764bb6246faac9ef4a992d3f999, State: Initialized, Role: FOLLOWER
I20260928 15:07:11.611310 13438 consensus_queue.cc:260] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.611627 13438 ts_tablet_manager.cc:1428] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:11.611634 13151 heartbeater.cc:503] Master 127.25.123.62:37019 was elected leader, sending a full tablet report...
I20260928 15:07:11.611689 13439 raft_consensus.cc:348] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.611878 13439 raft_consensus.cc:374] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:11.611927 13439 raft_consensus.cc:729] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd937607981343478892fe119f9ad1d1, State: Initialized, Role: FOLLOWER
I20260928 15:07:11.612040 13439 consensus_queue.cc:260] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.612803 13286 heartbeater.cc:503] Master 127.25.123.62:37019 was elected leader, sending a full tablet report...
I20260928 15:07:11.612813 13439 ts_tablet_manager.cc:1428] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:11.613431 13440 raft_consensus.cc:348] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.613571 13440 raft_consensus.cc:374] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:11.613595 13440 raft_consensus.cc:729] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddab053d8dd446fa2623f0f941931c0, State: Initialized, Role: FOLLOWER
I20260928 15:07:11.613731 13440 consensus_queue.cc:260] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.614071 13421 heartbeater.cc:503] Master 127.25.123.62:37019 was elected leader, sending a full tablet report...
I20260928 15:07:11.614073 13440 ts_tablet_manager.cc:1428] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:07:11.624140 13446 raft_consensus.cc:482] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:11.624333 13446 raft_consensus.cc:504] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.624923 13446 leader_election.cc:288] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd937607981343478892fe119f9ad1d1 (127.25.123.2:34477), fb5a3764bb6246faac9ef4a992d3f999 (127.25.123.1:38221)
I20260928 15:07:11.628996 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.625170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:41725 (local address 127.25.123.2:34477)
0928 15:07:11.625211 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.625213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.625566 (+ 353us) server_negotiation.cc:407] Connection header received
0928 15:07:11.625676 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.625679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.625689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.625745 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:11.625872 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.626516 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.627874 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.628576 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.628645 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.628652 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.628870 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.628921 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:07:11.628994 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:38221 (local address 127.25.123.3:47345)
0928 15:07:11.625128 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.625140 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.625179 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.625338 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.625340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.625342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:11.625430 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.625432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.626391 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.626392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.627747 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.627750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.628656 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.628657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.628752 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.628758 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.628824 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.628880 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":54}
I20260928 15:07:11.629001 13447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:34477 (local address 127.25.123.3:41725)
0928 15:07:11.625410 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.625472 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:11.625601 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:11.625748 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:11.625749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:11.625750 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:11.625811 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.625812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.626598 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.626599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.627747 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:11.627750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.628656 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.628657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:11.628751 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.628758 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:07:11.628825 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 15:07:11.628880 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":152,"thread_start_us":132,"threads_started":1}
I20260928 15:07:11.628996 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.625153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:47345 (local address 127.25.123.1:38221)
0928 15:07:11.625203 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.625205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.625216 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:11.625247 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.625248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.625259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.625335 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:11.625513 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.626290 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.627877 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.628530 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.628612 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.628625 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.628870 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.628921 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 15:07:11.629386 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9effb7c5ed404a88ba94dac0d58deb33" candidate_uuid: "cddab053d8dd446fa2623f0f941931c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5a3764bb6246faac9ef4a992d3f999" is_pre_election: true
I20260928 15:07:11.629536 13105 raft_consensus.cc:2454] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddab053d8dd446fa2623f0f941931c0 in term 0.
I20260928 15:07:11.629740 13240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9effb7c5ed404a88ba94dac0d58deb33" candidate_uuid: "cddab053d8dd446fa2623f0f941931c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd937607981343478892fe119f9ad1d1" is_pre_election: true
I20260928 15:07:11.629773 13308 leader_election.cc:302] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cddab053d8dd446fa2623f0f941931c0, fb5a3764bb6246faac9ef4a992d3f999; no voters:
I20260928 15:07:11.629895 13240 raft_consensus.cc:2454] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddab053d8dd446fa2623f0f941931c0 in term 0.
I20260928 15:07:11.629968 13446 raft_consensus.cc:2781] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:11.630049 13446 raft_consensus.cc:482] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:11.630080 13446 raft_consensus.cc:3037] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:11.630896 13446 raft_consensus.cc:504] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.631058 13446 leader_election.cc:288] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [CANDIDATE]: Term 1 election: Requested vote from peers dd937607981343478892fe119f9ad1d1 (127.25.123.2:34477), fb5a3764bb6246faac9ef4a992d3f999 (127.25.123.1:38221)
I20260928 15:07:11.631222 13240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9effb7c5ed404a88ba94dac0d58deb33" candidate_uuid: "cddab053d8dd446fa2623f0f941931c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd937607981343478892fe119f9ad1d1"
I20260928 15:07:11.631251 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9effb7c5ed404a88ba94dac0d58deb33" candidate_uuid: "cddab053d8dd446fa2623f0f941931c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5a3764bb6246faac9ef4a992d3f999"
I20260928 15:07:11.631336 13240 raft_consensus.cc:3037] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:11.631330 13105 raft_consensus.cc:3037] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:11.632092 13105 raft_consensus.cc:2454] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddab053d8dd446fa2623f0f941931c0 in term 1.
I20260928 15:07:11.632102 13240 raft_consensus.cc:2454] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddab053d8dd446fa2623f0f941931c0 in term 1.
I20260928 15:07:11.632269 13308 leader_election.cc:302] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cddab053d8dd446fa2623f0f941931c0, fb5a3764bb6246faac9ef4a992d3f999; no voters:
I20260928 15:07:11.632418 13446 raft_consensus.cc:2781] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:11.632575 13446 raft_consensus.cc:686] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 1 LEADER]: Becoming Leader. State: Replica: cddab053d8dd446fa2623f0f941931c0, State: Running, Role: LEADER
I20260928 15:07:11.632688 13446 consensus_queue.cc:237] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } }
I20260928 15:07:11.633514 12957 catalog_manager.cc:5495] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cddab053d8dd446fa2623f0f941931c0 (127.25.123.3). New cstate: current_term: 1 leader_uuid: "cddab053d8dd446fa2623f0f941931c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddab053d8dd446fa2623f0f941931c0" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 35337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 } health_report { overall_health: UNKNOWN } } }
W20260928 15:07:11.636536 13287 tablet.cc:2369] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:07:11.649014 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.646095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38616 (local address 127.25.123.3:35337)
0928 15:07:11.646167 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.646169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.646180 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:11.646221 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.646222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.646239 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.646303 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:11.646549 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.647094 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.647967 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.648172 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.648240 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.648561 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:11.648672 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:11.648745 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.648843 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.648874 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:07:11.651294 13105 raft_consensus.cc:1261] T 9effb7c5ed404a88ba94dac0d58deb33 P fb5a3764bb6246faac9ef4a992d3f999 [term 1 FOLLOWER]: Refusing update from remote peer cddab053d8dd446fa2623f0f941931c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:11.651424 13240 raft_consensus.cc:1261] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 1 FOLLOWER]: Refusing update from remote peer cddab053d8dd446fa2623f0f941931c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:11.651818 13448 consensus_queue.cc:1035] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5a3764bb6246faac9ef4a992d3f999" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 38221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:11.651985 13446 consensus_queue.cc:1035] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd937607981343478892fe119f9ad1d1" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:11.655030 13451 mvcc.cc:205] Tried to move back new op lower bound from 7334330497639591936 to 7334330497568280576. Current Snapshot: MvccSnapshot[applied={T|T < 7334330497639591936}]
I20260928 15:07:11.655504 13452 mvcc.cc:205] Tried to move back new op lower bound from 7334330497639591936 to 7334330497568280576. Current Snapshot: MvccSnapshot[applied={T|T < 7334330497639591936}]
I20260928 15:07:11.657522 12958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:11.657619 12958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:11.658478 12958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 460d8ca7e40142a09257bea288f4a681: Sending DeleteTablet for 3 replicas of tablet 9effb7c5ed404a88ba94dac0d58deb33
I20260928 15:07:11.658847 13220 tablet_service.cc:1514] Processing DeleteTablet for tablet 9effb7c5ed404a88ba94dac0d58deb33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:11 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:42246
I20260928 15:07:11.658842 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 9effb7c5ed404a88ba94dac0d58deb33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:11 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:47834
I20260928 15:07:11.659098 13461 tablet_replica.cc:330] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: stopping tablet replica
I20260928 15:07:11.659219 13461 raft_consensus.cc:2229] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:11.659296 13354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9effb7c5ed404a88ba94dac0d58deb33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:11 UTC) from {username='kudu', principal='kudu/127.25.123.62@KRBTEST.COM'} at 127.0.0.1:38602
I20260928 15:07:11.659332 13461 raft_consensus.cc:2258] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:11.659574 13463 tablet_replica.cc:330] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: stopping tablet replica
I20260928 15:07:11.659664 13463 raft_consensus.cc:2229] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:11.659711 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 13021
I20260928 15:07:11.660187 13461 ts_tablet_manager.cc:1905] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:11.659770 13463 raft_consensus.cc:2258] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:11.661717 13463 ts_tablet_manager.cc:1905] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:11.663479 13461 ts_tablet_manager.cc:1918] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:11.663627 13461 log.cc:1196] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-1/wal/wals/9effb7c5ed404a88ba94dac0d58deb33
I20260928 15:07:11.663879 13463 ts_tablet_manager.cc:1918] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:11.663969 13463 log.cc:1196] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/9effb7c5ed404a88ba94dac0d58deb33
I20260928 15:07:11.664091 13461 ts_tablet_manager.cc:1939] T 9effb7c5ed404a88ba94dac0d58deb33 P dd937607981343478892fe119f9ad1d1: Deleting consensus metadata
I20260928 15:07:11.664345 13463 ts_tablet_manager.cc:1939] T 9effb7c5ed404a88ba94dac0d58deb33 P cddab053d8dd446fa2623f0f941931c0: Deleting consensus metadata
I20260928 15:07:11.664806 12945 catalog_manager.cc:4841] TS dd937607981343478892fe119f9ad1d1 (127.25.123.2:34477): tablet 9effb7c5ed404a88ba94dac0d58deb33 (table test-table [id=69b284d9e0f3471090368b7e8d5540a6]) successfully deleted
I20260928 15:07:11.665087 12942 catalog_manager.cc:4841] TS cddab053d8dd446fa2623f0f941931c0 (127.25.123.3:35337): tablet 9effb7c5ed404a88ba94dac0d58deb33 (table test-table [id=69b284d9e0f3471090368b7e8d5540a6]) successfully deleted
W20260928 15:07:11.675689 12943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:38221 (error 108)
I20260928 15:07:11.675906 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 13156
I20260928 15:07:11.676973 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.676831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:38221 (local address 127.0.0.1:47846)
0928 15:07:11.676888 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:11.676926 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 15:07:11.677124 12943 catalog_manager.cc:4584] TS fb5a3764bb6246faac9ef4a992d3f999 (127.25.123.1:38221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9effb7c5ed404a88ba94dac0d58deb33: Network error: Client connection negotiation failed: client connection to 127.25.123.1:38221: connect: Connection refused (error 111)
I20260928 15:07:11.682024 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 13291
I20260928 15:07:11.687829 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 12926
2026-09-28T15:07:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2031 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T15:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:11Z Disabled control of system clock
I20260928 15:07:11.718354 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37949
--webserver_interface=127.25.123.62
--webserver_port=0
--builtin_ntp_servers=127.25.123.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:37949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:11.806772 13468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:11.807052 13468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:11.807078 13468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:11.808373 13468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:11.808483 13468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:11.808496 13468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:11.808509 13468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:11.808522 13468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:11.810047 13468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:37949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:11.810356 13468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:11.810705 13468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:11.816252 13474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.816252 13476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.816303 13473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:11.816732 13468 server_base.cc:1034] running on GCE node
I20260928 15:07:11.817009 13468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:11.817356 13468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:11.818538 13468 hybrid_clock.cc:648] HybridClock initialized: now 1790608031818527 us; error 35 us; skew 500 ppm
I20260928 15:07:11.820611 13468 webserver.cc:460] Webserver started at http://127.25.123.62:37291/ using document root <none> and password file <none>
I20260928 15:07:11.820894 13468 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:11.820968 13468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:11.821128 13468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:11.822296 13468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "273a30141f8840dcbf0b3cbdfc041e83"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "e16c82ed8654808fc6c51dbed2d77a42"
server_key_iv: "e134b33dd42a159ac13fe859ebc9972f"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.822702 13468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "273a30141f8840dcbf0b3cbdfc041e83"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "e16c82ed8654808fc6c51dbed2d77a42"
server_key_iv: "e134b33dd42a159ac13fe859ebc9972f"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.824183 13468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:11.825232 13482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:11.825582 13468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:11.825719 13468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "273a30141f8840dcbf0b3cbdfc041e83"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "e16c82ed8654808fc6c51dbed2d77a42"
server_key_iv: "e134b33dd42a159ac13fe859ebc9972f"
server_key_version: "encryptionkey@0"
I20260928 15:07:11.825809 13468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:11.837404 13468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:11.837790 13468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:11.837957 13468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:11.842705 13468 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:37949
I20260928 15:07:11.842752 13535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:37949 every 8 connection(s)
I20260928 15:07:11.843149 13468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:11.844043 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 13468
I20260928 15:07:11.844178 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:11.844384 26092 external_mini_cluster.cc:1428] Setting key cb46a8c7ac7eaaa5ecef3794f8fd5068
I20260928 15:07:11.844478 13536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:11.848244 13536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83: Bootstrap starting.
I20260928 15:07:11.848968 13536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:11.849289 13536 log.cc:824] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:11.850059 13536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83: No bootstrap required, opened a new log
I20260928 15:07:11.851308 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:11.845206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.25.123.62:37949)
0928 15:07:11.845442 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:11.845452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:11.845489 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:07:11.846284 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:11.846293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:11.846384 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:11.846556 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:11.846774 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.848159 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.849116 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:11.849615 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:11.849753 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:11.850659 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:11.850664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:11.850664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:11.850832 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:11.850866 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:11.850878 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:11.850960 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:11.851044 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 15:07:11.851134 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260928 15:07:11.852360 13536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a30141f8840dcbf0b3cbdfc041e83" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37949 } }
I20260928 15:07:11.852667 13536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:11.852721 13536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273a30141f8840dcbf0b3cbdfc041e83, State: Initialized, Role: FOLLOWER
I20260928 15:07:11.852895 13536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a30141f8840dcbf0b3cbdfc041e83" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37949 } }
I20260928 15:07:11.852993 13536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:11.853055 13536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:11.853109 13536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:11.853752 13536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a30141f8840dcbf0b3cbdfc041e83" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37949 } }
I20260928 15:07:11.853893 13536 leader_election.cc:302] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273a30141f8840dcbf0b3cbdfc041e83; no voters:
I20260928 15:07:11.854120 13536 leader_election.cc:288] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:11.854174 13541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:11.854372 13541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [term 1 LEADER]: Becoming Leader. State: Replica: 273a30141f8840dcbf0b3cbdfc041e83, State: Running, Role: LEADER
I20260928 15:07:11.854450 13536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:11.854466 13541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a30141f8840dcbf0b3cbdfc041e83" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37949 } }
I20260928 15:07:11.855139 13542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273a30141f8840dcbf0b3cbdfc041e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a30141f8840dcbf0b3cbdfc041e83" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37949 } } }
I20260928 15:07:11.855150 13543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273a30141f8840dcbf0b3cbdfc041e83. Latest consensus state: current_term: 1 leader_uuid: "273a30141f8840dcbf0b3cbdfc041e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a30141f8840dcbf0b3cbdfc041e83" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37949 } } }
I20260928 15:07:11.855279 13542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:11.855288 13543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:11.855535 13550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:11.856050 13550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:11.857873 13550 catalog_manager.cc:1347] Generated new cluster ID: ca19c3b47f274346b8e931deb1318078
I20260928 15:07:11.857944 13550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:11.861148 13550 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:11.862010 13550 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:11.879536 13550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83: Generated new TSK 0
I20260928 15:07:11.879798 13550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:11.884243 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.1:0
--local_ip_for_outbound_sockets=127.25.123.1
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37949
--builtin_ntp_servers=127.25.123.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:11.987116 13560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:11.987385 13560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:11.987428 13560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:11.988751 13560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:11.988816 13560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:11.988866 13560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.1
I20260928 15:07:11.990391 13560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.123.1
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.1
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:11.990674 13560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:11.990994 13560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:11.997335 13566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.997339 13565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:11.997335 13568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:11.998107 13560 server_base.cc:1034] running on GCE node
I20260928 15:07:11.998365 13560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:11.998651 13560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:11.999845 13560 hybrid_clock.cc:648] HybridClock initialized: now 1790608031999831 us; error 33 us; skew 500 ppm
I20260928 15:07:12.001564 13560 webserver.cc:460] Webserver started at http://127.25.123.1:36161/ using document root <none> and password file <none>
I20260928 15:07:12.001853 13560 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:12.001909 13560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:12.002089 13560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:12.003208 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data/instance:
uuid: "11ae89d5bd9846e0a58fb4ec37e179a9"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "149c0bee13cf219e032493d1b2985ec6"
server_key_iv: "12a246b11c45ccee74274f9e98ec66dc"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.003592 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance:
uuid: "11ae89d5bd9846e0a58fb4ec37e179a9"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "149c0bee13cf219e032493d1b2985ec6"
server_key_iv: "12a246b11c45ccee74274f9e98ec66dc"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.005125 13560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:07:12.006060 13574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.006268 13560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:12.006382 13560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/wal
uuid: "11ae89d5bd9846e0a58fb4ec37e179a9"
format_stamp: "Formatted at 2026-09-28 15:07:11 on dist-test-slave-qnz0"
server_key: "149c0bee13cf219e032493d1b2985ec6"
server_key_iv: "12a246b11c45ccee74274f9e98ec66dc"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.006507 13560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:12.021800 13560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:12.022183 13560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:12.022367 13560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:12.022647 13560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:12.023047 13560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:12.023087 13560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.023115 13560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:12.023130 13560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.029778 13560 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.1:40925
I20260928 15:07:12.029860 13687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.1:40925 every 8 connection(s)
I20260928 15:07:12.030257 13560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/data/info.pb
I20260928 15:07:12.030750 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 13560
I20260928 15:07:12.030874 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-0/wal/instance
I20260928 15:07:12.031070 26092 external_mini_cluster.cc:1428] Setting key 3eb621c439e50bb4290eb9fb98b274ec
I20260928 15:07:12.032341 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.2:0
--local_ip_for_outbound_sockets=127.25.123.2
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37949
--builtin_ntp_servers=127.25.123.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:07:12.037220 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.031268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.1:46697 (local address 127.25.123.62:37949)
0928 15:07:12.031326 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.031329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.031757 (+ 428us) server_negotiation.cc:407] Connection header received
0928 15:07:12.032354 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.032355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.032372 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.032493 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:12.033484 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.034015 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.035048 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.035248 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.035723 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.036852 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:12.036854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:12.036854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:12.036860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:12.036875 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:12.036886 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:12.036951 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.037097 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.037131 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:07:12.037447 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.031288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37949 (local address 127.25.123.1:46697)
0928 15:07:12.031560 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.031602 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.032226 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.032591 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.032597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.032608 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:12.033371 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.033378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.034129 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.034131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.034942 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.034945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.035732 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.035733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.035982 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.036682 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:12.036683 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:12.036712 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:12.036737 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:12.036953 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:12.036960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:12.036963 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.037098 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.037211 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260928 15:07:12.037971 13688 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37949
I20260928 15:07:12.038095 13688 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:12.038327 13688 heartbeater.cc:511] Master 127.25.123.62:37949 requested a full tablet report, sending...
I20260928 15:07:12.039004 13499 ts_manager.cc:194] Registered new tserver with Master: 11ae89d5bd9846e0a58fb4ec37e179a9 (127.25.123.1:40925)
I20260928 15:07:12.039773 13499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.1:46697
W20260928 15:07:12.116508 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:12.116734 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:12.116758 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:12.118088 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:12.118144 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:12.118172 13691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.2
I20260928 15:07:12.119596 13691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.123.2
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.2
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:12.119880 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:12.120203 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:12.125898 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:12.125898 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:12.126037 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:12.126269 13691 server_base.cc:1034] running on GCE node
I20260928 15:07:12.126492 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:12.126824 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:12.128093 13691 hybrid_clock.cc:648] HybridClock initialized: now 1790608032128063 us; error 62 us; skew 500 ppm
I20260928 15:07:12.129949 13691 webserver.cc:460] Webserver started at http://127.25.123.2:42215/ using document root <none> and password file <none>
I20260928 15:07:12.130236 13691 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:12.130288 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:12.130452 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:12.131906 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data/instance:
uuid: "b5bea7d12ec94650841ddd72c0a6959c"
format_stamp: "Formatted at 2026-09-28 15:07:12 on dist-test-slave-qnz0"
server_key: "54aad3e26a7b58aa952c06d69497b514"
server_key_iv: "fa26f36efd50c2900d28650375819bc3"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.132325 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance:
uuid: "b5bea7d12ec94650841ddd72c0a6959c"
format_stamp: "Formatted at 2026-09-28 15:07:12 on dist-test-slave-qnz0"
server_key: "54aad3e26a7b58aa952c06d69497b514"
server_key_iv: "fa26f36efd50c2900d28650375819bc3"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.133841 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:12.134711 13705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.134884 13691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:12.134984 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/wal
uuid: "b5bea7d12ec94650841ddd72c0a6959c"
format_stamp: "Formatted at 2026-09-28 15:07:12 on dist-test-slave-qnz0"
server_key: "54aad3e26a7b58aa952c06d69497b514"
server_key_iv: "fa26f36efd50c2900d28650375819bc3"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.135036 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:12.153281 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:12.153631 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:12.154090 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:12.154399 13691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:12.154789 13691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:12.154832 13691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.154888 13691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:12.154918 13691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.160864 13691 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.2:38081
I20260928 15:07:12.160949 13818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.2:38081 every 8 connection(s)
I20260928 15:07:12.161294 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/data/info.pb
I20260928 15:07:12.166983 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.162154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.2:52597 (local address 127.25.123.62:37949)
0928 15:07:12.162247 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.162249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.162489 (+ 240us) server_negotiation.cc:407] Connection header received
0928 15:07:12.163010 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.163011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.163022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.163077 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:12.163801 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.164319 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.165461 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.165649 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.165704 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.166641 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:12.166642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:12.166642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:12.166648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:12.166667 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:12.166679 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:12.166734 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.166896 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.166919 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:07:12.167212 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.162108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37949 (local address 127.25.123.2:52597)
0928 15:07:12.162348 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.162390 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.162912 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.163177 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.163181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.163190 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:12.163695 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.163699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.164531 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.164532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.165316 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.165319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.165757 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.165758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.165892 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.166486 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:12.166486 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:12.166515 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:12.166539 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:12.166774 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:12.166781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:12.166784 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.166878 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.166973 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":124,"threads_started":1}
I20260928 15:07:12.167936 13819 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37949
I20260928 15:07:12.168088 13819 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:12.168285 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 13691
I20260928 15:07:12.168310 13819 heartbeater.cc:511] Master 127.25.123.62:37949 requested a full tablet report, sending...
I20260928 15:07:12.168479 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-1/wal/instance
I20260928 15:07:12.168712 26092 external_mini_cluster.cc:1428] Setting key 7e80f9c840517280bf062cfcbebd9f3e
I20260928 15:07:12.168908 13499 ts_manager.cc:194] Registered new tserver with Master: b5bea7d12ec94650841ddd72c0a6959c (127.25.123.2:38081)
I20260928 15:07:12.169498 13499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.2:52597
I20260928 15:07:12.170009 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.123.3:0
--local_ip_for_outbound_sockets=127.25.123.3
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37949
--builtin_ntp_servers=127.25.123.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:07:12.256698 13822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:12.256963 13822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:12.257004 13822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:12.258209 13822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:12.258298 13822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:12.258349 13822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.123.3
I20260928 15:07:12.259835 13822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.123.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.123.3
--webserver_port=0
--tserver_master_addrs=127.25.123.62:37949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.123.3
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:12.260134 13822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:12.260501 13822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:12.266690 13827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:12.266640 13828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:12.266640 13830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:12.267038 13822 server_base.cc:1034] running on GCE node
I20260928 15:07:12.267263 13822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:12.267534 13822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:12.268750 13822 hybrid_clock.cc:648] HybridClock initialized: now 1790608032268727 us; error 39 us; skew 500 ppm
I20260928 15:07:12.270184 13822 webserver.cc:460] Webserver started at http://127.25.123.3:44495/ using document root <none> and password file <none>
I20260928 15:07:12.270512 13822 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:12.270594 13822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:12.270750 13822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:12.271891 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data/instance:
uuid: "3e653e8be6634127a50787da4ee308df"
format_stamp: "Formatted at 2026-09-28 15:07:12 on dist-test-slave-qnz0"
server_key: "5b4ab18c837ff704c45223e4dcdbe071"
server_key_iv: "a878b5b320be856e698dc6cba76c692b"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.272272 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance:
uuid: "3e653e8be6634127a50787da4ee308df"
format_stamp: "Formatted at 2026-09-28 15:07:12 on dist-test-slave-qnz0"
server_key: "5b4ab18c837ff704c45223e4dcdbe071"
server_key_iv: "a878b5b320be856e698dc6cba76c692b"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.274451 13822 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 15:07:12.275527 13836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.275910 13822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:12.276122 13822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal
uuid: "3e653e8be6634127a50787da4ee308df"
format_stamp: "Formatted at 2026-09-28 15:07:12 on dist-test-slave-qnz0"
server_key: "5b4ab18c837ff704c45223e4dcdbe071"
server_key_iv: "a878b5b320be856e698dc6cba76c692b"
server_key_version: "encryptionkey@0"
I20260928 15:07:12.276229 13822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:12.292954 13822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:12.293354 13822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:12.293535 13822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:12.293812 13822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:07:12.294175 13822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:07:12.294234 13822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.294276 13822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:07:12.294306 13822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:12.300259 13822 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.3:33937
I20260928 15:07:12.300303 13950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.3:33937 every 8 connection(s)
I20260928 15:07:12.300671 13822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/data/info.pb
I20260928 15:07:12.306104 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 13822
I20260928 15:07:12.306257 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal/instance
I20260928 15:07:12.306524 26092 external_mini_cluster.cc:1428] Setting key 71609ba6a955dd2eee7809cef6f1ca5b
I20260928 15:07:12.307277 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.301676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:34229 (local address 127.25.123.62:37949)
0928 15:07:12.301822 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.301832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.302183 (+ 351us) server_negotiation.cc:407] Connection header received
0928 15:07:12.302891 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.302892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.302907 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.302972 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:12.303772 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.304323 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.305458 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.305709 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.305798 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.306905 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:12.306907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:12.306908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:12.306915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:12.306939 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:12.306944 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:12.307002 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.307152 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.307184 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:07:12.307595 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.301525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:37949 (local address 127.25.123.3:34229)
0928 15:07:12.301908 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.301974 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.302705 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.303070 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.303082 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.303093 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:12.303653 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.303657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.304458 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.304459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.305347 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.305349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.305825 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.305826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.305988 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.306701 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:07:12.306702 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:12.306735 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:07:12.306762 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:12.307043 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:12.307050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:12.307054 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.307157 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.307300 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260928 15:07:12.308324 13951 heartbeater.cc:347] Connected to a master server at 127.25.123.62:37949
I20260928 15:07:12.308490 13951 heartbeater.cc:464] Registering TS with master...
I20260928 15:07:12.308699 13951 heartbeater.cc:511] Master 127.25.123.62:37949 requested a full tablet report, sending...
I20260928 15:07:12.309244 13499 ts_manager.cc:194] Registered new tserver with Master: 3e653e8be6634127a50787da4ee308df (127.25.123.3:33937)
I20260928 15:07:12.309669 13499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.123.3:34229
I20260928 15:07:12.318976 26092 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:07:12.323741 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.320680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.25.123.62:37949)
0928 15:07:12.320760 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.320762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.320874 (+ 112us) server_negotiation.cc:407] Connection header received
0928 15:07:12.320903 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.320903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.320914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.320972 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:12.321159 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.321682 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.322554 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.322741 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.322806 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.323505 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:12.323506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:12.323506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:12.323511 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:12.323525 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:12.323529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:12.323575 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.323651 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.323673 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:07:12.325851 13499 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:07:12.327661 13499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:07:12.335207 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.330080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.3:33937 (local address 127.0.0.1:36682)
0928 15:07:12.330442 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.330488 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.330565 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.330930 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.330931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.330933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:12.330991 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.330993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.332334 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.332335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.334092 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.334096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.334869 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.334870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.334950 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.334965 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.335050 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.335111 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":235,"thread_start_us":64,"threads_started":1}
I20260928 15:07:12.335640 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.330263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.25.123.3:33937)
0928 15:07:12.330720 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.330723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.330733 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:07:12.330771 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.330773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.330790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.330863 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:12.331068 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.332249 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.334220 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.334758 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.334864 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.334968 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.335068 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.335110 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":95,"threads_started":1}
I20260928 15:07:12.335726 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.330079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40925 (local address 127.0.0.1:35926)
0928 15:07:12.330443 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.330487 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.330566 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.330860 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.330863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.330865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:12.330941 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.330943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.332551 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.332551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.334149 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.334153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.335247 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.335248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.335318 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.335324 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.335375 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.335501 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":36,"thread_start_us":69,"threads_started":1}
I20260928 15:07:12.335856 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.330429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.25.123.2:38081)
0928 15:07:12.330629 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.330634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.331067 (+ 433us) server_negotiation.cc:407] Connection header received
0928 15:07:12.331111 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.331113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.331127 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.331223 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:12.331313 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.332041 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.334273 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.335060 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.335126 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.335208 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.335673 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.335728 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260928 15:07:12.336072 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.330427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.25.123.1:40925)
0928 15:07:12.330682 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.330686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.330696 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:07:12.330737 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.330739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.330754 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.330854 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:12.331095 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.332249 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.334273 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.335052 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.335126 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.335209 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.335946 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.335994 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":142,"threads_started":1}
I20260928 15:07:12.336480 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.330287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:38081 (local address 127.0.0.1:52230)
0928 15:07:12.330948 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.330959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.331005 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.331221 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.331222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.331223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:12.331258 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.331260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.332248 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.332250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.334150 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.334153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.335248 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.335249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.335318 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.335324 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.335375 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.335503 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":20,"thread_start_us":70,"threads_started":1}
I20260928 15:07:12.336681 13884 tablet_service.cc:1467] Processing CreateTablet for tablet fbf00db2e4024ab9a3d25b93f84d7f46 (DEFAULT_TABLE table=test-table [id=bc0d2e29c3294169b5db5e9d35d0e70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:12.336803 13622 tablet_service.cc:1467] Processing CreateTablet for tablet fbf00db2e4024ab9a3d25b93f84d7f46 (DEFAULT_TABLE table=test-table [id=bc0d2e29c3294169b5db5e9d35d0e70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:12.337131 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf00db2e4024ab9a3d25b93f84d7f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:12.337152 13622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf00db2e4024ab9a3d25b93f84d7f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:12.337196 13753 tablet_service.cc:1467] Processing CreateTablet for tablet fbf00db2e4024ab9a3d25b93f84d7f46 (DEFAULT_TABLE table=test-table [id=bc0d2e29c3294169b5db5e9d35d0e70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:07:12.337524 13753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf00db2e4024ab9a3d25b93f84d7f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:12.341228 13967 tablet_bootstrap.cc:492] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9: Bootstrap starting.
I20260928 15:07:12.341377 13966 tablet_bootstrap.cc:492] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Bootstrap starting.
I20260928 15:07:12.342057 13966 tablet_bootstrap.cc:654] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:12.342059 13967 tablet_bootstrap.cc:654] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:12.342372 13966 log.cc:824] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:12.342373 13967 log.cc:824] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:12.342763 13965 tablet_bootstrap.cc:492] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: Bootstrap starting.
I20260928 15:07:12.343186 13966 tablet_bootstrap.cc:492] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: No bootstrap required, opened a new log
I20260928 15:07:12.343228 13967 tablet_bootstrap.cc:492] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9: No bootstrap required, opened a new log
I20260928 15:07:12.343288 13967 ts_tablet_manager.cc:1397] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:07:12.343287 13966 ts_tablet_manager.cc:1397] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:07:12.343508 13965 tablet_bootstrap.cc:654] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:12.343935 13965 log.cc:824] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:12.344897 13965 tablet_bootstrap.cc:492] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: No bootstrap required, opened a new log
I20260928 15:07:12.344978 13965 ts_tablet_manager.cc:1397] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:07:12.346114 13966 raft_consensus.cc:348] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.346338 13966 raft_consensus.cc:374] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:12.346386 13966 raft_consensus.cc:729] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e653e8be6634127a50787da4ee308df, State: Initialized, Role: FOLLOWER
I20260928 15:07:12.346522 13966 consensus_queue.cc:260] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.346513 13967 raft_consensus.cc:348] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.346701 13967 raft_consensus.cc:374] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:12.346743 13967 raft_consensus.cc:729] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ae89d5bd9846e0a58fb4ec37e179a9, State: Initialized, Role: FOLLOWER
I20260928 15:07:12.346851 13966 ts_tablet_manager.cc:1428] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 15:07:12.346885 13967 consensus_queue.cc:260] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.346985 13951 heartbeater.cc:503] Master 127.25.123.62:37949 was elected leader, sending a full tablet report...
I20260928 15:07:12.347201 13967 ts_tablet_manager.cc:1428] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:07:12.347227 13688 heartbeater.cc:503] Master 127.25.123.62:37949 was elected leader, sending a full tablet report...
I20260928 15:07:12.348009 13965 raft_consensus.cc:348] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.348199 13965 raft_consensus.cc:374] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:12.348246 13965 raft_consensus.cc:729] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bea7d12ec94650841ddd72c0a6959c, State: Initialized, Role: FOLLOWER
I20260928 15:07:12.348383 13965 consensus_queue.cc:260] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.348721 13965 ts_tablet_manager.cc:1428] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 15:07:12.348802 13819 heartbeater.cc:503] Master 127.25.123.62:37949 was elected leader, sending a full tablet report...
I20260928 15:07:12.349884 13971 raft_consensus.cc:482] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:07:12.349969 13971 raft_consensus.cc:504] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.350315 13971 leader_election.cc:288] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11ae89d5bd9846e0a58fb4ec37e179a9 (127.25.123.1:40925), b5bea7d12ec94650841ddd72c0a6959c (127.25.123.2:38081)
I20260928 15:07:12.353829 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.350461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40925 (local address 127.25.123.3:51557)
0928 15:07:12.350533 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.350548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.350604 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.350741 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.350742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.350743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:12.350837 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.350839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.351663 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.351664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.352690 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.352694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.353410 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.353412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.353488 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.353494 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.353659 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.353721 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 15:07:12.353827 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.350532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:51557 (local address 127.25.123.1:40925)
0928 15:07:12.350577 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.350579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.350589 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:07:12.350664 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.350665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.350676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.350737 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:12.350911 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.351579 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.352776 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.353220 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.353278 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.353285 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.353675 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.353751 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:07:12.354105 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.350583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:38081 (local address 127.25.123.3:58561)
0928 15:07:12.350734 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.350745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:12.350801 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:12.350944 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:12.350945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:12.350946 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:07:12.351018 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.351019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.351945 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.351946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.352840 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:12.352843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.353748 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.353749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:12.353832 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.353837 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:07:12.353902 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 15:07:12.354002 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":107,"threads_started":1}
I20260928 15:07:12.354272 13642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf00db2e4024ab9a3d25b93f84d7f46" candidate_uuid: "3e653e8be6634127a50787da4ee308df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" is_pre_election: true
I20260928 15:07:12.354403 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.350777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.123.3:58561 (local address 127.25.123.2:38081)
0928 15:07:12.350827 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.350829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.350838 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:12.350861 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.350862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.350874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.350938 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:07:12.351131 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.351859 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.352944 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.353643 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.353721 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.353730 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.354280 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.354337 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:07:12.354486 13642 raft_consensus.cc:2454] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e653e8be6634127a50787da4ee308df in term 0.
I20260928 15:07:12.354665 13773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf00db2e4024ab9a3d25b93f84d7f46" candidate_uuid: "3e653e8be6634127a50787da4ee308df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bea7d12ec94650841ddd72c0a6959c" is_pre_election: true
I20260928 15:07:12.354714 13837 leader_election.cc:302] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ae89d5bd9846e0a58fb4ec37e179a9, 3e653e8be6634127a50787da4ee308df; no voters:
I20260928 15:07:12.354844 13971 raft_consensus.cc:2781] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:07:12.354867 13773 raft_consensus.cc:2454] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e653e8be6634127a50787da4ee308df in term 0.
I20260928 15:07:12.354902 13971 raft_consensus.cc:482] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:07:12.354933 13971 raft_consensus.cc:3037] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:12.355863 13971 raft_consensus.cc:504] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.356004 13971 leader_election.cc:288] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [CANDIDATE]: Term 1 election: Requested vote from peers 11ae89d5bd9846e0a58fb4ec37e179a9 (127.25.123.1:40925), b5bea7d12ec94650841ddd72c0a6959c (127.25.123.2:38081)
I20260928 15:07:12.356164 13642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf00db2e4024ab9a3d25b93f84d7f46" candidate_uuid: "3e653e8be6634127a50787da4ee308df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9"
I20260928 15:07:12.356235 13773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf00db2e4024ab9a3d25b93f84d7f46" candidate_uuid: "3e653e8be6634127a50787da4ee308df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bea7d12ec94650841ddd72c0a6959c"
I20260928 15:07:12.356253 13642 raft_consensus.cc:3037] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:12.356316 13773 raft_consensus.cc:3037] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:12.357214 13773 raft_consensus.cc:2454] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e8be6634127a50787da4ee308df in term 1.
I20260928 15:07:12.357214 13642 raft_consensus.cc:2454] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e653e8be6634127a50787da4ee308df in term 1.
I20260928 15:07:12.357429 13837 leader_election.cc:302] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ae89d5bd9846e0a58fb4ec37e179a9, 3e653e8be6634127a50787da4ee308df; no voters:
I20260928 15:07:12.357559 13971 raft_consensus.cc:2781] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:12.357712 13971 raft_consensus.cc:686] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 1 LEADER]: Becoming Leader. State: Replica: 3e653e8be6634127a50787da4ee308df, State: Running, Role: LEADER
I20260928 15:07:12.357823 13971 consensus_queue.cc:237] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } }
I20260928 15:07:12.358884 13499 catalog_manager.cc:5495] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e653e8be6634127a50787da4ee308df (127.25.123.3). New cstate: current_term: 1 leader_uuid: "3e653e8be6634127a50787da4ee308df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e653e8be6634127a50787da4ee308df" member_type: VOTER last_known_addr { host: "127.25.123.3" port: 33937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 } health_report { overall_health: UNKNOWN } } }
I20260928 15:07:12.371017 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.368282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.25.123.3:33937)
0928 15:07:12.368384 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:12.368386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:12.368650 (+ 264us) server_negotiation.cc:407] Connection header received
0928 15:07:12.368688 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:12.368689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:12.368708 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:12.368792 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:12.368952 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.369431 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.370200 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:12.370396 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:12.370483 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:12.370647 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:12.370760 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:12.370852 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:12.370875 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 15:07:12.370895 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:07:12.373250 13773 raft_consensus.cc:1261] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 1 FOLLOWER]: Refusing update from remote peer 3e653e8be6634127a50787da4ee308df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:12.373250 13642 raft_consensus.cc:1261] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 1 FOLLOWER]: Refusing update from remote peer 3e653e8be6634127a50787da4ee308df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:12.373656 13971 consensus_queue.cc:1035] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ae89d5bd9846e0a58fb4ec37e179a9" member_type: VOTER last_known_addr { host: "127.25.123.1" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:12.373828 13975 consensus_queue.cc:1035] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5bea7d12ec94650841ddd72c0a6959c" member_type: VOTER last_known_addr { host: "127.25.123.2" port: 38081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:12.377197 13979 mvcc.cc:205] Tried to move back new op lower bound from 7334330500597043200 to 7334330500538593280. Current Snapshot: MvccSnapshot[applied={T|T < 7334330500597043200}]
I20260928 15:07:12.377887 13980 mvcc.cc:205] Tried to move back new op lower bound from 7334330500597043200 to 7334330500538593280. Current Snapshot: MvccSnapshot[applied={T|T < 7334330500597043200}]
I20260928 15:07:12.379756 13498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:12.379863 13498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:07:12.380856 13498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 273a30141f8840dcbf0b3cbdfc041e83: Sending DeleteTablet for 3 replicas of tablet fbf00db2e4024ab9a3d25b93f84d7f46
I20260928 15:07:12.381263 13884 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf00db2e4024ab9a3d25b93f84d7f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:12 UTC) from {username='slave'} at 127.0.0.1:36682
I20260928 15:07:12.381259 13753 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf00db2e4024ab9a3d25b93f84d7f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:12 UTC) from {username='slave'} at 127.0.0.1:52230
I20260928 15:07:12.381532 13988 tablet_replica.cc:330] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: stopping tablet replica
I20260928 15:07:12.381536 13622 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf00db2e4024ab9a3d25b93f84d7f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:07:12 UTC) from {username='slave'} at 127.0.0.1:35926
I20260928 15:07:12.381644 13988 raft_consensus.cc:2229] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:07:12.381778 13988 raft_consensus.cc:2258] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:12.381794 13990 tablet_replica.cc:330] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9: stopping tablet replica
I20260928 15:07:12.381985 13990 raft_consensus.cc:2229] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:12.382158 13990 raft_consensus.cc:2258] T fbf00db2e4024ab9a3d25b93f84d7f46 P 11ae89d5bd9846e0a58fb4ec37e179a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:12.382175 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 13560
I20260928 15:07:12.382965 13988 ts_tablet_manager.cc:1905] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:07:12.385193 13988 ts_tablet_manager.cc:1918] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:07:12.385281 13988 log.cc:1196] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Deleting WAL directory at /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790607748072058-26092-0/minicluster-data/ts-2/wal/wals/fbf00db2e4024ab9a3d25b93f84d7f46
I20260928 15:07:12.385542 13988 ts_tablet_manager.cc:1939] T fbf00db2e4024ab9a3d25b93f84d7f46 P 3e653e8be6634127a50787da4ee308df: Deleting consensus metadata
I20260928 15:07:12.386231 13484 catalog_manager.cc:4841] TS 3e653e8be6634127a50787da4ee308df (127.25.123.3:33937): tablet fbf00db2e4024ab9a3d25b93f84d7f46 (table test-table [id=bc0d2e29c3294169b5db5e9d35d0e70e]) successfully deleted
I20260928 15:07:12.388622 13989 tablet_replica.cc:330] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: stopping tablet replica
I20260928 15:07:12.388792 13989 raft_consensus.cc:2229] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:07:12.388936 13989 raft_consensus.cc:2258] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:07:12.389873 13989 ts_tablet_manager.cc:1905] T fbf00db2e4024ab9a3d25b93f84d7f46 P b5bea7d12ec94650841ddd72c0a6959c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 15:07:12.390287 13483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.123.1:40925 (error 108)
W20260928 15:07:12.390847 13959 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:12.390681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.1:40925 (local address 127.0.0.1:35936)
0928 15:07:12.390749 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.390794 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.1:40925: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 15:07:12.390975 13483 catalog_manager.cc:4584] TS 11ae89d5bd9846e0a58fb4ec37e179a9 (127.25.123.1:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbf00db2e4024ab9a3d25b93f84d7f46: Network error: Client connection negotiation failed: client connection to 127.25.123.1:40925: connect: Connection reset by peer (error 104)
I20260928 15:07:12.391321 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 13691
I20260928 15:07:12.396870 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 13822
I20260928 15:07:12.397073 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:12.396846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.2:38081 (local address 127.0.0.1:52234)
0928 15:07:12.396981 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:12.397031 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.2:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 15:07:12.403530 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 13468
2026-09-28T15:07:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (713 ms)
[----------] 6 tests from EncryptionPolicyTest (8844 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:12 dist-test-slave-qnz0 krb5kdc[13994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:12 dist-test-slave-qnz0 krb5kdc[13994](info): set up 2 sockets
Sep 28 15:07:12 dist-test-slave-qnz0 krb5kdc[13994](info): commencing operation
krb5kdc: starting...
W20260928 15:07:13.484591 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:13 dist-test-slave-qnz0 krb5kdc[13994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:13.613639 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:13.703073 14007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:13.703290 14007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:13.703315 14007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:13.704571 14007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:13.704622 14007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:13.704638 14007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:13.704650 14007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:13.704662 14007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:13.706182 14007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:13.706390 14007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:13.706573 14007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:13.711800 14013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:13.711833 14007 server_base.cc:1034] running on GCE node
W20260928 15:07:13.711776 14012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:13.711797 14015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:13.712447 14007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:13.712757 14007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:13.713930 14007 hybrid_clock.cc:648] HybridClock initialized: now 1790608033713922 us; error 42 us; skew 500 ppm
Sep 28 15:07:13 dist-test-slave-qnz0 krb5kdc[13994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:13.716638 14007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:13.717139 14007 webserver.cc:460] Webserver started at http://127.0.0.1:34303/ using document root <none> and password file <none>
I20260928 15:07:13.717393 14007 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:13.717439 14007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:13.717583 14007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:13.718703 14007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "9698451a430d4db08a20c91f2e912f04"
format_stamp: "Formatted at 2026-09-28 15:07:13 on dist-test-slave-qnz0"
server_key: "e6df351952054ad056f95e6fd44a0902"
server_key_iv: "e0df2432d467fc4359bc8d0f6f2bf0cd"
server_key_version: "encryptionkey@0"
I20260928 15:07:13.719066 14007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "9698451a430d4db08a20c91f2e912f04"
format_stamp: "Formatted at 2026-09-28 15:07:13 on dist-test-slave-qnz0"
server_key: "e6df351952054ad056f95e6fd44a0902"
server_key_iv: "e0df2432d467fc4359bc8d0f6f2bf0cd"
server_key_version: "encryptionkey@0"
I20260928 15:07:13.720480 14007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 15:07:13.721287 14022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:13.721576 14007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:13.721664 14007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "9698451a430d4db08a20c91f2e912f04"
format_stamp: "Formatted at 2026-09-28 15:07:13 on dist-test-slave-qnz0"
server_key: "e6df351952054ad056f95e6fd44a0902"
server_key_iv: "e0df2432d467fc4359bc8d0f6f2bf0cd"
server_key_version: "encryptionkey@0"
I20260928 15:07:13.721714 14007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:13.735805 14007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:13.738200 14007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:13.738318 14007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:13.742373 14007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39489
I20260928 15:07:13.742472 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39489 every 8 connection(s)
I20260928 15:07:13.742815 14007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:13.743780 14076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:13.746148 14076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04: Bootstrap starting.
I20260928 15:07:13.746747 14076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:13.747042 14076 log.cc:824] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:13.747925 14076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04: No bootstrap required, opened a new log
I20260928 15:07:13.749446 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14007
I20260928 15:07:13.749610 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:13.749729 14076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698451a430d4db08a20c91f2e912f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39489 } }
I20260928 15:07:13.749862 26092 external_mini_cluster.cc:1428] Setting key ccf51f33782f60fa7cd37445fe602328
I20260928 15:07:13.749900 14076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:13.749951 14076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9698451a430d4db08a20c91f2e912f04, State: Initialized, Role: FOLLOWER
I20260928 15:07:13.750118 14076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698451a430d4db08a20c91f2e912f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39489 } }
I20260928 15:07:13.750213 14076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:13.750252 14076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:13.750327 14076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:13.751464 14076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698451a430d4db08a20c91f2e912f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39489 } }
I20260928 15:07:13.751559 14076 leader_election.cc:302] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9698451a430d4db08a20c91f2e912f04; no voters:
I20260928 15:07:13.751720 14076 leader_election.cc:288] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:13.751744 14081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:13.751853 14081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [term 1 LEADER]: Becoming Leader. State: Replica: 9698451a430d4db08a20c91f2e912f04, State: Running, Role: LEADER
I20260928 15:07:13.751948 14081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698451a430d4db08a20c91f2e912f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39489 } }
I20260928 15:07:13.751991 14076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:13.752264 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9698451a430d4db08a20c91f2e912f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698451a430d4db08a20c91f2e912f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39489 } } }
I20260928 15:07:13.752328 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:13.752470 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9698451a430d4db08a20c91f2e912f04. Latest consensus state: current_term: 1 leader_uuid: "9698451a430d4db08a20c91f2e912f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9698451a430d4db08a20c91f2e912f04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39489 } } }
I20260928 15:07:13.752547 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:13.752687 14085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:13.753278 14085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:13.756431 14085 catalog_manager.cc:1347] Generated new cluster ID: 35e8e735f5ee46ffaffca2c450e07594
I20260928 15:07:13.756495 14085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:07:13 dist-test-slave-qnz0 krb5kdc[13994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:13.761301 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:13.750462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46485 (local address 127.0.0.1:39489)
0928 15:07:13.750699 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:13.750709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:13.750744 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:07:13.750881 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:13.750888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:13.750970 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:13.751152 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:13.752074 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:13.753481 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:13.754430 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:13.755026 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:13.755146 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:13.757798 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:13.757802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:13.757802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:13.757814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:13.759552 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:13.759970 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:13.759978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:13.759978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:13.760080 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:13.760318 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:13.760320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:13.760320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:13.760698 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:13.760895 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:13.760995 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 15:07:13.761100 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":120,"threads_started":1}
I20260928 15:07:13.762264 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:13.776032 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:13.764719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37941 (local address 127.0.0.1:39489)
0928 15:07:13.764766 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:13.764768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:13.768684 (+ 3916us) server_negotiation.cc:407] Connection header received
0928 15:07:13.768781 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:13.768782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:13.768794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:13.768906 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:13.769565 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:13.770130 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:13.770961 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:13.771220 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:13.771310 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:13.772798 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:13.772799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:13.772800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:13.772808 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:13.774446 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:13.775102 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:13.775104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:13.775104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:13.775174 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:13.775422 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:13.775423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:13.775424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:13.775565 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:13.775710 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:13.775824 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 15:07:13.775893 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":22}
I20260928 15:07:13.778563 14085 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:13.779120 14085 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:13.782984 14085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9698451a430d4db08a20c91f2e912f04: Generated new TSK 0
I20260928 15:07:13.783200 14085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:13.787685 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14007
W20260928 15:07:13.792382 13992 connection.cc:535] client connection to 127.0.0.1:39489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39489): Connection reset by peer (error 104)
2026-09-28T15:07:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:13 dist-test-slave-qnz0 krb5kdc[14107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:13 dist-test-slave-qnz0 krb5kdc[14107](info): set up 2 sockets
Sep 28 15:07:13 dist-test-slave-qnz0 krb5kdc[14107](info): commencing operation
krb5kdc: starting...
W20260928 15:07:14.879570 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:14 dist-test-slave-qnz0 krb5kdc[14107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:15.012669 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:15.096774 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:15.097026 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:15.097064 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:15.098259 14120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:15.098392 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:15.098428 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:15.098460 14120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:15.098483 14120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:15.100016 14120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:15.100317 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:15.100589 14120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:15.106334 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:15.106362 14120 server_base.cc:1034] running on GCE node
W20260928 15:07:15.106334 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:15.106369 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:15.106928 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:15.107381 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:15.108589 14120 hybrid_clock.cc:648] HybridClock initialized: now 1790608035108567 us; error 42 us; skew 500 ppm
Sep 28 15:07:15 dist-test-slave-qnz0 krb5kdc[14107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:15.117470 14120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:15.118069 14120 webserver.cc:460] Webserver started at http://127.0.0.1:39535/ using document root <none> and password file <none>
I20260928 15:07:15.118315 14120 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:15.118388 14120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:15.118552 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:15.119719 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "e0182734a1594e02ac258c67a59f8bb5"
format_stamp: "Formatted at 2026-09-28 15:07:15 on dist-test-slave-qnz0"
server_key: "af470285c3c75372cabac963b39a96d5"
server_key_iv: "fe913f8b0cd654b6d2042a06396fa50f"
server_key_version: "encryptionkey@0"
I20260928 15:07:15.120141 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "e0182734a1594e02ac258c67a59f8bb5"
format_stamp: "Formatted at 2026-09-28 15:07:15 on dist-test-slave-qnz0"
server_key: "af470285c3c75372cabac963b39a96d5"
server_key_iv: "fe913f8b0cd654b6d2042a06396fa50f"
server_key_version: "encryptionkey@0"
I20260928 15:07:15.121665 14120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:07:15.122498 14135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:15.122677 14120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:15.122784 14120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "e0182734a1594e02ac258c67a59f8bb5"
format_stamp: "Formatted at 2026-09-28 15:07:15 on dist-test-slave-qnz0"
server_key: "af470285c3c75372cabac963b39a96d5"
server_key_iv: "fe913f8b0cd654b6d2042a06396fa50f"
server_key_version: "encryptionkey@0"
I20260928 15:07:15.122874 14120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:15.149380 14120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:15.151568 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:15.151746 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:15.156041 14120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35779
I20260928 15:07:15.156076 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35779 every 8 connection(s)
I20260928 15:07:15.156536 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:15.157506 14189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:15.158442 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14120
I20260928 15:07:15.158551 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:15.158761 26092 external_mini_cluster.cc:1428] Setting key 856d28afe9ed7958e090e34999b0bcff
I20260928 15:07:15.160192 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5: Bootstrap starting.
I20260928 15:07:15.160893 14189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:15.161183 14189 log.cc:824] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:15.161936 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5: No bootstrap required, opened a new log
I20260928 15:07:15.164309 14189 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182734a1594e02ac258c67a59f8bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 15:07:15.164585 14189 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:15.164640 14189 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0182734a1594e02ac258c67a59f8bb5, State: Initialized, Role: FOLLOWER
I20260928 15:07:15.164762 14189 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182734a1594e02ac258c67a59f8bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 15:07:15.164805 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:15.159292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39599 (local address 127.0.0.1:35779)
0928 15:07:15.159454 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:15.159461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:15.159487 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:07:15.159955 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:15.159961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:15.160006 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:15.160130 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:15.160378 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:15.161438 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:15.162304 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:15.162975 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:15.163104 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:15.163881 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:15.163888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:15.163888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:15.164122 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:15.164162 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:15.164178 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:15.164270 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:15.164540 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 15:07:15.164571 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260928 15:07:15.164855 14189 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:15.164891 14189 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:15.164933 14189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:15.165664 14189 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182734a1594e02ac258c67a59f8bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 15:07:15.165776 14189 leader_election.cc:302] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0182734a1594e02ac258c67a59f8bb5; no voters:
I20260928 15:07:15.165884 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:15.165951 14189 leader_election.cc:288] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:15.166005 14194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:15.166143 14194 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [term 1 LEADER]: Becoming Leader. State: Replica: e0182734a1594e02ac258c67a59f8bb5, State: Running, Role: LEADER
I20260928 15:07:15.166224 14194 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182734a1594e02ac258c67a59f8bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 15:07:15.166246 14189 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:15.166569 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0182734a1594e02ac258c67a59f8bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182734a1594e02ac258c67a59f8bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } } }
I20260928 15:07:15.166690 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:15.167014 14199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:15.167560 14199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:15.168008 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0182734a1594e02ac258c67a59f8bb5. Latest consensus state: current_term: 1 leader_uuid: "e0182734a1594e02ac258c67a59f8bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0182734a1594e02ac258c67a59f8bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } } }
I20260928 15:07:15.168118 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:15.169483 14199 catalog_manager.cc:1347] Generated new cluster ID: cf4ac9d6941841f58ff341079a8ac6a6
I20260928 15:07:15.169540 14199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:15.171644 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:15.167354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59563 (local address 127.0.0.1:35779)
0928 15:07:15.167389 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:15.167391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:15.168186 (+ 795us) server_negotiation.cc:407] Connection header received
0928 15:07:15.168275 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:15.168276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:15.168287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:15.168354 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:15.168558 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:15.169245 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:15.170261 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:15.170569 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:15.170648 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:15.171384 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:15.171385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:15.171386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:15.171391 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:15.171408 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:15.171413 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:15.171479 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:15.171558 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 15:07:15.171569 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:07:15.183118 14199 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:15.183730 14199 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:15.188031 14199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0182734a1594e02ac258c67a59f8bb5: Generated new TSK 0
I20260928 15:07:15.188254 14199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:15.191170 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14120
W20260928 15:07:15.195722 14105 connection.cc:535] client connection to 127.0.0.1:35779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35779): Connection reset by peer (error 104)
2026-09-28T15:07:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:15 dist-test-slave-qnz0 krb5kdc[14221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:15 dist-test-slave-qnz0 krb5kdc[14221](info): set up 2 sockets
Sep 28 15:07:15 dist-test-slave-qnz0 krb5kdc[14221](info): commencing operation
krb5kdc: starting...
W20260928 15:07:16.247283 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:16 dist-test-slave-qnz0 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:16.383730 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:16.465929 14234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:16.466198 14234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:16.466228 14234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:16.467706 14234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:16.467768 14234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:16.467784 14234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:16.467799 14234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:16.467813 14234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:16.469624 14234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:16.469937 14234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:16.470221 14234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:16.476471 14242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:16.476487 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:16.476465 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:16.476745 14234 server_base.cc:1034] running on GCE node
I20260928 15:07:16.477030 14234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:16.477324 14234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:16.478564 14234 hybrid_clock.cc:648] HybridClock initialized: now 1790608036478540 us; error 35 us; skew 500 ppm
Sep 28 15:07:16 dist-test-slave-qnz0 krb5kdc[14221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:16.481556 14234 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:16.482097 14234 webserver.cc:460] Webserver started at http://127.0.0.1:44983/ using document root <none> and password file <none>
I20260928 15:07:16.482360 14234 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:16.482411 14234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:16.482577 14234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:16.483834 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "2c43c7a5f5f6415191087c364b7c16c4"
format_stamp: "Formatted at 2026-09-28 15:07:16 on dist-test-slave-qnz0"
server_key: "7a2531ddb82eb08f37caf8f30e4f49d8"
server_key_iv: "ed31e8ff12c05990f0d79433b4cf3a8a"
server_key_version: "encryptionkey@0"
I20260928 15:07:16.484258 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "2c43c7a5f5f6415191087c364b7c16c4"
format_stamp: "Formatted at 2026-09-28 15:07:16 on dist-test-slave-qnz0"
server_key: "7a2531ddb82eb08f37caf8f30e4f49d8"
server_key_iv: "ed31e8ff12c05990f0d79433b4cf3a8a"
server_key_version: "encryptionkey@0"
I20260928 15:07:16.485925 14234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:16.486827 14249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:16.487088 14234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:16.487203 14234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "2c43c7a5f5f6415191087c364b7c16c4"
format_stamp: "Formatted at 2026-09-28 15:07:16 on dist-test-slave-qnz0"
server_key: "7a2531ddb82eb08f37caf8f30e4f49d8"
server_key_iv: "ed31e8ff12c05990f0d79433b4cf3a8a"
server_key_version: "encryptionkey@0"
I20260928 15:07:16.487300 14234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:16.491180 14234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:16.493853 14234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:16.494043 14234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:16.498616 14234 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40945
I20260928 15:07:16.498659 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40945 every 8 connection(s)
I20260928 15:07:16.499037 14234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:16.499372 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14234
I20260928 15:07:16.499497 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:16.499744 26092 external_mini_cluster.cc:1428] Setting key 500f1bf792049aa51de0d2d9246563f2
I20260928 15:07:16.500351 14303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:16.502722 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:16.500354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54889 (local address 127.0.0.1:40945)
0928 15:07:16.500607 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:16.500616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:16.500644 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:07:16.501168 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:16.501175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:16.501203 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:16.501348 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:16.502121 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:16.502124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:16.502125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:16.502129 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:16.502171 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:16.502183 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:16.502246 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:16.502462 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 15:07:16.502527 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":146,"threads_started":1}
I20260928 15:07:16.503741 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:16.504603 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4: Bootstrap starting.
I20260928 15:07:16.505530 14303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:16.505905 14303 log.cc:824] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:16.506707 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4: No bootstrap required, opened a new log
I20260928 15:07:16.507066 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:16.505623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56051 (local address 127.0.0.1:40945)
0928 15:07:16.505666 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:16.505668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:16.505710 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:07:16.505784 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:16.505785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:16.505791 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:16.505841 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:16.506733 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:16.506734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:16.506734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:16.506739 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:16.506764 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:16.506769 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:16.506852 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:16.506937 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 15:07:16.506951 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:07:16.508960 14303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43c7a5f5f6415191087c364b7c16c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40945 } }
I20260928 15:07:16.509135 14303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:16.509176 14303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c43c7a5f5f6415191087c364b7c16c4, State: Initialized, Role: FOLLOWER
I20260928 15:07:16.509307 14303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43c7a5f5f6415191087c364b7c16c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40945 } }
I20260928 15:07:16.509392 14303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:16.509413 14303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:16.509443 14303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:16.509992 14303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43c7a5f5f6415191087c364b7c16c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40945 } }
I20260928 15:07:16.510082 14303 leader_election.cc:302] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c43c7a5f5f6415191087c364b7c16c4; no voters:
I20260928 15:07:16.510227 14303 leader_election.cc:288] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:16.510293 14313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:16.510444 14313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [term 1 LEADER]: Becoming Leader. State: Replica: 2c43c7a5f5f6415191087c364b7c16c4, State: Running, Role: LEADER
I20260928 15:07:16.510504 14303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:16.510535 14313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43c7a5f5f6415191087c364b7c16c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40945 } }
I20260928 15:07:16.510919 14314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c43c7a5f5f6415191087c364b7c16c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43c7a5f5f6415191087c364b7c16c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40945 } } }
I20260928 15:07:16.511083 14314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:16.510933 14315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c43c7a5f5f6415191087c364b7c16c4. Latest consensus state: current_term: 1 leader_uuid: "2c43c7a5f5f6415191087c364b7c16c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43c7a5f5f6415191087c364b7c16c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40945 } } }
I20260928 15:07:16.511281 14315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:16.511541 14321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:16.512084 14321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:16.513845 14321 catalog_manager.cc:1347] Generated new cluster ID: e31f9a89ad47455d930ab2a9ae0d7a75
I20260928 15:07:16.513899 14321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:16.554539 14321 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:16.554773 14321 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:16.560307 14321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c43c7a5f5f6415191087c364b7c16c4: Generated new TSK 0
I20260928 15:07:16.560568 14321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:16.563045 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14234
2026-09-28T15:07:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:16 dist-test-slave-qnz0 krb5kdc[14334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:16 dist-test-slave-qnz0 krb5kdc[14334](info): set up 2 sockets
Sep 28 15:07:16 dist-test-slave-qnz0 krb5kdc[14334](info): commencing operation
krb5kdc: starting...
W20260928 15:07:17.644153 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:17 dist-test-slave-qnz0 krb5kdc[14334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:17.785763 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:17.868530 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:17.868804 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:17.868844 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:17.870029 14347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:17.870110 14347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:17.870136 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:17.870158 14347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:17.870188 14347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:17.871735 14347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:17.872053 14347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:17.872298 14347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:17.878615 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:17.878752 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:17.878789 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:17.879592 14347 server_base.cc:1034] running on GCE node
I20260928 15:07:17.879904 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:17.880198 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:17.881372 14347 hybrid_clock.cc:648] HybridClock initialized: now 1790608037881348 us; error 34 us; skew 500 ppm
Sep 28 15:07:17 dist-test-slave-qnz0 krb5kdc[14334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:17.884205 14347 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:17.884876 14347 webserver.cc:460] Webserver started at http://127.0.0.1:37017/ using document root <none> and password file <none>
I20260928 15:07:17.885154 14347 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:17.885216 14347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:17.885357 14347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:17.886492 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "b3f87d86a7f34b8a952f973bf11955ef"
format_stamp: "Formatted at 2026-09-28 15:07:17 on dist-test-slave-qnz0"
server_key: "19b2770c6b8a6bfd6b5b90c1f4d4c929"
server_key_iv: "171a302bb09ea744fea46499f9a2e93d"
server_key_version: "encryptionkey@0"
I20260928 15:07:17.886868 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "b3f87d86a7f34b8a952f973bf11955ef"
format_stamp: "Formatted at 2026-09-28 15:07:17 on dist-test-slave-qnz0"
server_key: "19b2770c6b8a6bfd6b5b90c1f4d4c929"
server_key_iv: "171a302bb09ea744fea46499f9a2e93d"
server_key_version: "encryptionkey@0"
I20260928 15:07:17.888383 14347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:17.889423 14362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:17.889822 14347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:17.889922 14347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "b3f87d86a7f34b8a952f973bf11955ef"
format_stamp: "Formatted at 2026-09-28 15:07:17 on dist-test-slave-qnz0"
server_key: "19b2770c6b8a6bfd6b5b90c1f4d4c929"
server_key_iv: "171a302bb09ea744fea46499f9a2e93d"
server_key_version: "encryptionkey@0"
I20260928 15:07:17.890051 14347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:17.904673 14347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:17.907338 14347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:17.907507 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:17.912106 14347 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35199
I20260928 15:07:17.912184 14414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35199 every 8 connection(s)
I20260928 15:07:17.912590 14347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:17.914026 14415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:17.916651 14415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef: Bootstrap starting.
I20260928 15:07:17.917316 14415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:17.917642 14415 log.cc:824] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:17.918536 14415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef: No bootstrap required, opened a new log
I20260928 15:07:17.920222 14415 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f87d86a7f34b8a952f973bf11955ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35199 } }
I20260928 15:07:17.920382 14415 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:17.920477 14415 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f87d86a7f34b8a952f973bf11955ef, State: Initialized, Role: FOLLOWER
I20260928 15:07:17.920702 14415 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f87d86a7f34b8a952f973bf11955ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35199 } }
I20260928 15:07:17.920782 14415 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:17.920805 14415 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:17.920854 14415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:17.921190 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14347
I20260928 15:07:17.921283 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:17.921483 26092 external_mini_cluster.cc:1428] Setting key 33985d2641a041d74171baebdefee303
I20260928 15:07:17.921540 14415 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f87d86a7f34b8a952f973bf11955ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35199 } }
I20260928 15:07:17.921657 14415 leader_election.cc:302] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f87d86a7f34b8a952f973bf11955ef; no voters:
I20260928 15:07:17.921847 14415 leader_election.cc:288] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:17.921927 14418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:17.922206 14415 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:17.922358 14418 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [term 1 LEADER]: Becoming Leader. State: Replica: b3f87d86a7f34b8a952f973bf11955ef, State: Running, Role: LEADER
I20260928 15:07:17.922447 14418 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f87d86a7f34b8a952f973bf11955ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35199 } }
I20260928 15:07:17.922842 14420 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f87d86a7f34b8a952f973bf11955ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f87d86a7f34b8a952f973bf11955ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35199 } } }
I20260928 15:07:17.922914 14420 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:17.923080 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f87d86a7f34b8a952f973bf11955ef. Latest consensus state: current_term: 1 leader_uuid: "b3f87d86a7f34b8a952f973bf11955ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f87d86a7f34b8a952f973bf11955ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35199 } } }
I20260928 15:07:17.923141 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:17.923477 14424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:17.924045 14424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:07:17 dist-test-slave-qnz0 krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:17.928884 14424 catalog_manager.cc:1347] Generated new cluster ID: b1212dae0d2845bda8d377773a39e1f6
I20260928 15:07:17.928944 14424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:17.934638 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:17.922013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49259 (local address 127.0.0.1:35199)
0928 15:07:17.922235 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:17.922245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:17.922291 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:07:17.922412 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:17.922420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:17.922476 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:17.922616 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:17.923569 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.925023 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.925802 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.926425 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.926541 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:17.929367 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:17.929371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:17.929372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:17.929384 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:17.932636 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.933094 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.933098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.933099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.933238 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.933463 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.933464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.933465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.933828 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:17.934008 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:17.934100 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 15:07:17.934382 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260928 15:07:17.935869 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:17.945181 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:17.937415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47139 (local address 127.0.0.1:35199)
0928 15:07:17.937504 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:17.937507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:17.937617 (+ 110us) server_negotiation.cc:407] Connection header received
0928 15:07:17.937650 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:17.937651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:17.937663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:17.937734 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:17.938334 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.939212 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.940230 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:17.940575 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:17.940653 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:17.942101 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:17.942102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:17.942103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:17.942110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:17.943753 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.944231 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.944233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.944233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.944296 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:17.944516 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:17.944517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:17.944518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:17.944669 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:17.944774 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:17.944890 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 15:07:17.944972 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:07:17.946306 14424 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:17.946810 14424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:17.955822 14424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f87d86a7f34b8a952f973bf11955ef: Generated new TSK 0
I20260928 15:07:17.956087 14424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:17.964082 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14347
2026-09-28T15:07:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:18 dist-test-slave-qnz0 krb5kdc[14446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:18 dist-test-slave-qnz0 krb5kdc[14446](info): set up 2 sockets
Sep 28 15:07:18 dist-test-slave-qnz0 krb5kdc[14446](info): commencing operation
krb5kdc: starting...
W20260928 15:07:19.063501 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:19 dist-test-slave-qnz0 krb5kdc[14446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:19.197597 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:19.290025 14459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:19.290282 14459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:19.290321 14459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:19.291527 14459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:19.291605 14459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:19.291630 14459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:19.291651 14459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:19.291682 14459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:19.293318 14459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:19.293759 14459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:19.294075 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:19.299731 14464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:19.299724 14467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:19.299723 14465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:19.299827 14459 server_base.cc:1034] running on GCE node
I20260928 15:07:19.300323 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:19.300733 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:19.301909 14459 hybrid_clock.cc:648] HybridClock initialized: now 1790608039301876 us; error 57 us; skew 500 ppm
Sep 28 15:07:19 dist-test-slave-qnz0 krb5kdc[14446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:19.304934 14459 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:19.305557 14459 webserver.cc:460] Webserver started at http://127.0.0.1:36057/ using document root <none> and password file <none>
I20260928 15:07:19.305836 14459 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:19.305887 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:19.306025 14459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:19.307221 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "63c71bcfc01243bdbde59bbbf9d9278e"
format_stamp: "Formatted at 2026-09-28 15:07:19 on dist-test-slave-qnz0"
server_key: "44fc70f1350b9c5f7122d4625ed2aaf7"
server_key_iv: "3a7f3aebadd66da0dd61b8ed06809976"
server_key_version: "encryptionkey@0"
I20260928 15:07:19.307608 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "63c71bcfc01243bdbde59bbbf9d9278e"
format_stamp: "Formatted at 2026-09-28 15:07:19 on dist-test-slave-qnz0"
server_key: "44fc70f1350b9c5f7122d4625ed2aaf7"
server_key_iv: "3a7f3aebadd66da0dd61b8ed06809976"
server_key_version: "encryptionkey@0"
I20260928 15:07:19.309255 14459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:19.310226 14474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:19.310660 14459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:19.310796 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "63c71bcfc01243bdbde59bbbf9d9278e"
format_stamp: "Formatted at 2026-09-28 15:07:19 on dist-test-slave-qnz0"
server_key: "44fc70f1350b9c5f7122d4625ed2aaf7"
server_key_iv: "3a7f3aebadd66da0dd61b8ed06809976"
server_key_version: "encryptionkey@0"
I20260928 15:07:19.310889 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:19.324112 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:19.327055 14459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:19.327251 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:19.332521 14459 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41579
I20260928 15:07:19.332602 14526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41579 every 8 connection(s)
I20260928 15:07:19.333155 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:19.334064 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14459
I20260928 15:07:19.334223 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:19.334460 26092 external_mini_cluster.cc:1428] Setting key 6ed65adb1f21b6755b08fe4874f880dd
I20260928 15:07:19.334566 14527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:19.337919 14527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e: Bootstrap starting.
I20260928 15:07:19.338579 14527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:19.338905 14527 log.cc:824] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:19.339695 14527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e: No bootstrap required, opened a new log
I20260928 15:07:19.340950 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.335035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37467 (local address 127.0.0.1:41579)
0928 15:07:19.335275 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.335291 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.335317 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:07:19.335975 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.335984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.336046 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.336335 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:19.336719 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.337700 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.338576 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.339108 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.339240 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.340177 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:19.340182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:19.340182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:19.340370 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:19.340461 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:19.340487 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:19.340563 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.340687 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.340765 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
I20260928 15:07:19.342000 14527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 15:07:19.342211 14527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:19.342240 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:19.342252 14527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c71bcfc01243bdbde59bbbf9d9278e, State: Initialized, Role: FOLLOWER
I20260928 15:07:19.342370 14527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 15:07:19.342444 14527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:19.342471 14527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:19.342523 14527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:19.343355 14527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 15:07:19.343604 14527 leader_election.cc:302] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63c71bcfc01243bdbde59bbbf9d9278e; no voters:
I20260928 15:07:19.343786 14527 leader_election.cc:288] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:19.343824 14536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:19.343964 14536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [term 1 LEADER]: Becoming Leader. State: Replica: 63c71bcfc01243bdbde59bbbf9d9278e, State: Running, Role: LEADER
I20260928 15:07:19.344177 14527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:19.344260 14536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260928 15:07:19.345422 14538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63c71bcfc01243bdbde59bbbf9d9278e. Latest consensus state: current_term: 1 leader_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } } }
I20260928 15:07:19.345513 14538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:19.345947 14537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c71bcfc01243bdbde59bbbf9d9278e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } } }
I20260928 15:07:19.346025 14537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e [sys.catalog]: This master's current role is: LEADER
W20260928 15:07:19.346238 14553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:19.346349 14553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:19.346446 14554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:19.346675 14554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:19.348201 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:19.343906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36869 (local address 127.0.0.1:41579)
0928 15:07:19.343964 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:19.343966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:19.344211 (+ 245us) server_negotiation.cc:407] Connection header received
0928 15:07:19.344300 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:19.344301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:19.344313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:19.344358 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:19.344542 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.345247 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.346353 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:19.346661 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:19.346721 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:19.347802 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:19.347810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:19.347810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:19.347815 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:19.347832 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:19.347836 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:19.348012 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:19.348092 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 15:07:19.348119 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:07:19.348985 14554 catalog_manager.cc:1347] Generated new cluster ID: a6c4f542de004c9fa712390aa346db68
I20260928 15:07:19.349038 14554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:19.353086 14554 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:19.353924 14554 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:19.362488 14554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63c71bcfc01243bdbde59bbbf9d9278e: Generated new TSK 0
I20260928 15:07:19.362732 14554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:19.367794 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14459
2026-09-28T15:07:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:19 dist-test-slave-qnz0 krb5kdc[14559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:19 dist-test-slave-qnz0 krb5kdc[14559](info): set up 2 sockets
Sep 28 15:07:19 dist-test-slave-qnz0 krb5kdc[14559](info): commencing operation
krb5kdc: starting...
W20260928 15:07:20.431577 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:20 dist-test-slave-qnz0 krb5kdc[14559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:20.568704 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:20.652505 14572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:20.652740 14572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:20.652765 14572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:20.653959 14572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:20.654000 14572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:20.654014 14572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:20.654027 14572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:20.654039 14572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:20.655843 14572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:20.656129 14572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:20.656427 14572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:20.662328 14580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:20.662320 14577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:20.662323 14578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:20.662623 14572 server_base.cc:1034] running on GCE node
I20260928 15:07:20.662911 14572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:20.663224 14572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:20.664470 14572 hybrid_clock.cc:648] HybridClock initialized: now 1790608040664430 us; error 37 us; skew 500 ppm
Sep 28 15:07:20 dist-test-slave-qnz0 krb5kdc[14559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:20.676970 14572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:20.677522 14572 webserver.cc:460] Webserver started at http://127.0.0.1:42021/ using document root <none> and password file <none>
I20260928 15:07:20.677757 14572 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:20.677822 14572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:20.677976 14572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:20.679128 14572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "68586082106d43399d380241cd3cb6a7"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-qnz0"
server_key: "10b5685b9f72b5565486b94f037ac17d"
server_key_iv: "ddbe363b04bbad99b4ecf1d017e899da"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.679509 14572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "68586082106d43399d380241cd3cb6a7"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-qnz0"
server_key: "10b5685b9f72b5565486b94f037ac17d"
server_key_iv: "ddbe363b04bbad99b4ecf1d017e899da"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.681087 14572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:20.681999 14587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:20.682250 14572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:20.682438 14572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "68586082106d43399d380241cd3cb6a7"
format_stamp: "Formatted at 2026-09-28 15:07:20 on dist-test-slave-qnz0"
server_key: "10b5685b9f72b5565486b94f037ac17d"
server_key_iv: "ddbe363b04bbad99b4ecf1d017e899da"
server_key_version: "encryptionkey@0"
I20260928 15:07:20.682632 14572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:20.686607 14572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:20.688668 14572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:20.688831 14572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:20.692948 14572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42793
I20260928 15:07:20.693012 14639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42793 every 8 connection(s)
I20260928 15:07:20.693365 14572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:20.693899 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14572
I20260928 15:07:20.694033 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:20.694205 26092 external_mini_cluster.cc:1428] Setting key 3a9f4271b5589f7c7eac93652950eb57
I20260928 15:07:20.694510 14640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:20.697301 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:20.694986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58559 (local address 127.0.0.1:42793)
0928 15:07:20.695179 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:20.695187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:20.695217 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:07:20.695882 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:20.695890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:20.695918 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:20.696104 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:20.696828 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:20.696832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:20.696832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:20.696836 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:20.696875 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:20.696885 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:20.696951 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:20.697053 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 15:07:20.697105 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260928 15:07:20.698271 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:20.698459 14640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7: Bootstrap starting.
I20260928 15:07:20.699203 14640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:20.699564 14640 log.cc:824] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:20.700428 14640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7: No bootstrap required, opened a new log
I20260928 15:07:20.701025 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:20.699946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58949 (local address 127.0.0.1:42793)
0928 15:07:20.699980 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:20.699982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:20.700116 (+ 134us) server_negotiation.cc:407] Connection header received
0928 15:07:20.700174 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:20.700175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:20.700181 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:20.700241 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:20.700820 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:20.700821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:20.700821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:20.700825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:20.700842 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:20.700846 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:20.700921 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:20.700955 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 15:07:20.700967 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 15:07:20.702788 14640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68586082106d43399d380241cd3cb6a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42793 } }
I20260928 15:07:20.702942 14640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:20.702968 14640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68586082106d43399d380241cd3cb6a7, State: Initialized, Role: FOLLOWER
I20260928 15:07:20.703123 14640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68586082106d43399d380241cd3cb6a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42793 } }
I20260928 15:07:20.703212 14640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:20.703235 14640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:20.703287 14640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:20.703995 14640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68586082106d43399d380241cd3cb6a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42793 } }
I20260928 15:07:20.704118 14640 leader_election.cc:302] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68586082106d43399d380241cd3cb6a7; no voters:
I20260928 15:07:20.704285 14640 leader_election.cc:288] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:20.704316 14651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:20.704583 14640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:20.704602 14651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [term 1 LEADER]: Becoming Leader. State: Replica: 68586082106d43399d380241cd3cb6a7, State: Running, Role: LEADER
I20260928 15:07:20.704695 14651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68586082106d43399d380241cd3cb6a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42793 } }
I20260928 15:07:20.705101 14653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68586082106d43399d380241cd3cb6a7. Latest consensus state: current_term: 1 leader_uuid: "68586082106d43399d380241cd3cb6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68586082106d43399d380241cd3cb6a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42793 } } }
I20260928 15:07:20.705367 14653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:20.705296 14652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68586082106d43399d380241cd3cb6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68586082106d43399d380241cd3cb6a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42793 } } }
I20260928 15:07:20.705593 14652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:20.705735 14660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:20.706266 14660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:20.707839 14660 catalog_manager.cc:1347] Generated new cluster ID: d55cb5eae3344ecb9640c87d28d43851
I20260928 15:07:20.707888 14660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:20.713590 14660 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:20.713775 14660 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:20.717859 14660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68586082106d43399d380241cd3cb6a7: Generated new TSK 0
I20260928 15:07:20.718058 14660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:20.727300 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14572
2026-09-28T15:07:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:20 dist-test-slave-qnz0 krb5kdc[14672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:20 dist-test-slave-qnz0 krb5kdc[14672](info): set up 2 sockets
Sep 28 15:07:20 dist-test-slave-qnz0 krb5kdc[14672](info): commencing operation
krb5kdc: starting...
W20260928 15:07:21.791424 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:21 dist-test-slave-qnz0 krb5kdc[14672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:21.927569 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:22.015532 14685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:22.015810 14685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:22.015847 14685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:22.017115 14685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:22.017187 14685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:22.017210 14685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:22.017232 14685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:22.017258 14685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:22.018850 14685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:22.019150 14685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:22.019394 14685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:22.024555 14693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:22.024619 14691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:22.024629 14685 server_base.cc:1034] running on GCE node
W20260928 15:07:22.024577 14690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:22.025045 14685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:22.025342 14685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:22.026506 14685 hybrid_clock.cc:648] HybridClock initialized: now 1790608042026492 us; error 43 us; skew 500 ppm
Sep 28 15:07:22 dist-test-slave-qnz0 krb5kdc[14672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:22.029184 14685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:22.029682 14685 webserver.cc:460] Webserver started at http://127.0.0.1:38851/ using document root <none> and password file <none>
I20260928 15:07:22.029876 14685 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:22.029919 14685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:22.030015 14685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:22.031119 14685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "79a69a3560ae4da08143d8351811a09b"
format_stamp: "Formatted at 2026-09-28 15:07:22 on dist-test-slave-qnz0"
server_key: "7298e42d15cda336331f6b9d091dbfd0"
server_key_iv: "05b5153c86d1c6d8e144867f2ce80fc6"
server_key_version: "encryptionkey@0"
I20260928 15:07:22.031427 14685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "79a69a3560ae4da08143d8351811a09b"
format_stamp: "Formatted at 2026-09-28 15:07:22 on dist-test-slave-qnz0"
server_key: "7298e42d15cda336331f6b9d091dbfd0"
server_key_iv: "05b5153c86d1c6d8e144867f2ce80fc6"
server_key_version: "encryptionkey@0"
I20260928 15:07:22.032835 14685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:07:22.033651 14700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:22.033857 14685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:22.033951 14685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "79a69a3560ae4da08143d8351811a09b"
format_stamp: "Formatted at 2026-09-28 15:07:22 on dist-test-slave-qnz0"
server_key: "7298e42d15cda336331f6b9d091dbfd0"
server_key_iv: "05b5153c86d1c6d8e144867f2ce80fc6"
server_key_version: "encryptionkey@0"
I20260928 15:07:22.034036 14685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:22.053911 14685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:22.056071 14685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:22.056259 14685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:22.060490 14685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40649
I20260928 15:07:22.060549 14752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40649 every 8 connection(s)
I20260928 15:07:22.061019 14685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:22.062014 14753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:22.062856 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14685
I20260928 15:07:22.062987 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:22.063205 26092 external_mini_cluster.cc:1428] Setting key 58b2ce073fe7891c193541b7233795fa
I20260928 15:07:22.065395 14753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b: Bootstrap starting.
I20260928 15:07:22.065999 14753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:22.066285 14753 log.cc:824] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:22.066967 14753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b: No bootstrap required, opened a new log
I20260928 15:07:22.068989 14753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a69a3560ae4da08143d8351811a09b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40649 } }
I20260928 15:07:22.069149 14753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:22.069197 14753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a69a3560ae4da08143d8351811a09b, State: Initialized, Role: FOLLOWER
I20260928 15:07:22.069375 14753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a69a3560ae4da08143d8351811a09b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40649 } }
I20260928 15:07:22.069470 14753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:22.069514 14753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:22.069561 14753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:22.070302 14753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a69a3560ae4da08143d8351811a09b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40649 } }
I20260928 15:07:22.070430 14753 leader_election.cc:302] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79a69a3560ae4da08143d8351811a09b; no voters:
Sep 28 15:07:22 dist-test-slave-qnz0 krb5kdc[14672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:22.070602 14753 leader_election.cc:288] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:22.070783 14759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:22.070885 14753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:22.071000 14759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [term 1 LEADER]: Becoming Leader. State: Replica: 79a69a3560ae4da08143d8351811a09b, State: Running, Role: LEADER
I20260928 15:07:22.071132 14759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a69a3560ae4da08143d8351811a09b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40649 } }
I20260928 15:07:22.071532 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79a69a3560ae4da08143d8351811a09b. Latest consensus state: current_term: 1 leader_uuid: "79a69a3560ae4da08143d8351811a09b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a69a3560ae4da08143d8351811a09b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40649 } } }
I20260928 15:07:22.071889 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:22.071794 14760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79a69a3560ae4da08143d8351811a09b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a69a3560ae4da08143d8351811a09b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40649 } } }
I20260928 15:07:22.072310 14760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b [sys.catalog]: This master's current role is: LEADER
W20260928 15:07:22.073061 14774 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:22.073128 14774 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:22.073180 14769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:22.073338 14769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:22.075731 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:22.063773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:38109 (local address 127.0.0.1:40649)
0928 15:07:22.063979 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:22.063990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:22.064024 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:07:22.064615 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:22.064624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:22.064693 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:22.064859 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:22.065609 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.066667 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.067519 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.067955 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.068089 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:22.071159 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:22.071164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:22.071165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:22.071178 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:22.073682 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:22.074269 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:22.074272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:22.074273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:22.074386 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:22.074728 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:22.074729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:22.074729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:22.075065 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:22.075256 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:22.075394 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 15:07:22.075511 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260928 15:07:22.076879 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:22.076929 14769 catalog_manager.cc:1347] Generated new cluster ID: 282574c73ae044bf8e871ac241cea0e8
I20260928 15:07:22.076968 14769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:22.086565 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:22.078375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:53129 (local address 127.0.0.1:40649)
0928 15:07:22.078414 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:22.078416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:22.078715 (+ 299us) server_negotiation.cc:407] Connection header received
0928 15:07:22.078767 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:22.078767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:22.078779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:22.078855 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:22.079501 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.080385 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.081492 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:22.081836 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:22.081915 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:22.083450 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:22.083451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:22.083451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:22.083464 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:22.085155 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:22.085650 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:22.085652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:22.085652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:22.085726 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:22.085964 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:22.085965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:22.085965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:22.086116 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:22.086221 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:22.086376 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:07:22.086459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:07:22.099592 14769 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:22.100232 14769 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:22.104725 14769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79a69a3560ae4da08143d8351811a09b: Generated new TSK 0
I20260928 15:07:22.105012 14769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:22.111994 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14685
W20260928 15:07:22.116950 14670 connection.cc:535] client connection to 127.0.0.1:40649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40649): Connection reset by peer (error 104)
2026-09-28T15:07:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:22 dist-test-slave-qnz0 krb5kdc[14784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:22 dist-test-slave-qnz0 krb5kdc[14784](info): set up 2 sockets
Sep 28 15:07:22 dist-test-slave-qnz0 krb5kdc[14784](info): commencing operation
krb5kdc: starting...
W20260928 15:07:23.175340 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:23 dist-test-slave-qnz0 krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:23.307701 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:23.391914 14797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:23.392148 14797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:23.392172 14797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:23.393463 14797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:23.393517 14797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:23.393532 14797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:23.393544 14797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:23.393558 14797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:23.395119 14797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:23.395401 14797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:23.395641 14797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:23.401264 14803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:23.401341 14797 server_base.cc:1034] running on GCE node
W20260928 15:07:23.401273 14802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:23.401336 14805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:23.401858 14797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:23.402226 14797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:23.403414 14797 hybrid_clock.cc:648] HybridClock initialized: now 1790608043403398 us; error 38 us; skew 500 ppm
Sep 28 15:07:23 dist-test-slave-qnz0 krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:23.406016 14797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:23.406549 14797 webserver.cc:460] Webserver started at http://127.0.0.1:39829/ using document root <none> and password file <none>
I20260928 15:07:23.406790 14797 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:23.406857 14797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:23.407008 14797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:23.408100 14797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "0c77de306031434abc3b2a700cb682ee"
format_stamp: "Formatted at 2026-09-28 15:07:23 on dist-test-slave-qnz0"
server_key: "21b3e126a9837c6d714dd61618643787"
server_key_iv: "cbed4a19222cccd84383eb4c8df72ee8"
server_key_version: "encryptionkey@0"
I20260928 15:07:23.408540 14797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "0c77de306031434abc3b2a700cb682ee"
format_stamp: "Formatted at 2026-09-28 15:07:23 on dist-test-slave-qnz0"
server_key: "21b3e126a9837c6d714dd61618643787"
server_key_iv: "cbed4a19222cccd84383eb4c8df72ee8"
server_key_version: "encryptionkey@0"
I20260928 15:07:23.410032 14797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:23.410846 14812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:23.411020 14797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 15:07:23.411121 14797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "0c77de306031434abc3b2a700cb682ee"
format_stamp: "Formatted at 2026-09-28 15:07:23 on dist-test-slave-qnz0"
server_key: "21b3e126a9837c6d714dd61618643787"
server_key_iv: "cbed4a19222cccd84383eb4c8df72ee8"
server_key_version: "encryptionkey@0"
I20260928 15:07:23.411207 14797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:23.439491 14797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:23.442261 14797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:23.442543 14797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:23.446835 14797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32901
I20260928 15:07:23.446923 14865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32901 every 8 connection(s)
I20260928 15:07:23.447279 14797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:23.448257 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:23.450992 14866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee: Bootstrap starting.
I20260928 15:07:23.451622 14866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:23.451920 14866 log.cc:824] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:23.452884 14866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee: No bootstrap required, opened a new log
I20260928 15:07:23.453388 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14797
I20260928 15:07:23.453511 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:23.453701 26092 external_mini_cluster.cc:1428] Setting key 0b99cb0c83a956475b67fc3c324e1dad
I20260928 15:07:23.455554 14866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c77de306031434abc3b2a700cb682ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260928 15:07:23.455751 14866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:23.455828 14866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c77de306031434abc3b2a700cb682ee, State: Initialized, Role: FOLLOWER
I20260928 15:07:23.456001 14866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c77de306031434abc3b2a700cb682ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260928 15:07:23.456113 14866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:23.456153 14866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:23.456200 14866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:23.457139 14866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c77de306031434abc3b2a700cb682ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260928 15:07:23.457275 14866 leader_election.cc:302] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c77de306031434abc3b2a700cb682ee; no voters:
I20260928 15:07:23.457443 14866 leader_election.cc:288] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:23.457513 14871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:23.457702 14866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:23.457736 14871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [term 1 LEADER]: Becoming Leader. State: Replica: 0c77de306031434abc3b2a700cb682ee, State: Running, Role: LEADER
I20260928 15:07:23.457964 14871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c77de306031434abc3b2a700cb682ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260928 15:07:23.458400 14872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c77de306031434abc3b2a700cb682ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c77de306031434abc3b2a700cb682ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } } }
I20260928 15:07:23.458381 14873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c77de306031434abc3b2a700cb682ee. Latest consensus state: current_term: 1 leader_uuid: "0c77de306031434abc3b2a700cb682ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c77de306031434abc3b2a700cb682ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } } }
I20260928 15:07:23.458496 14873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:23.458495 14872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:23.458803 14880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:23.459416 14880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:23.461810 14880 catalog_manager.cc:1347] Generated new cluster ID: 16aa72ac5be7482bbc100e6f7eeb4ef0
I20260928 15:07:23.461872 14880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:23.461894 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:23.454410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:33091 (local address 127.0.0.1:32901)
0928 15:07:23.454650 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:23.454659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:23.454684 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:07:23.455546 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:23.455555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:23.455612 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:23.455787 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:23.456070 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:23.457826 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:23.459150 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:23.459925 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:23.460048 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:23.460956 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:23.460961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:23.460961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:23.461201 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:23.461242 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:23.461259 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:23.461548 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:23.461645 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 15:07:23.461676 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":136,"threads_started":1}
I20260928 15:07:23.462978 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:23.468453 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:23.464534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:47601 (local address 127.0.0.1:32901)
0928 15:07:23.464570 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:23.464572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:23.464706 (+ 134us) server_negotiation.cc:407] Connection header received
0928 15:07:23.464759 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:23.464760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:23.464791 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:23.464849 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:23.465001 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:23.465820 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:23.466895 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:23.467265 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:23.467330 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:23.468136 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:23.468137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:23.468138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:23.468143 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:23.468158 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:23.468163 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:23.468217 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:23.468310 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 15:07:23.468324 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:07:23.472957 14880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:23.473774 14880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:23.479235 14880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c77de306031434abc3b2a700cb682ee: Generated new TSK 0
I20260928 15:07:23.479451 14880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:23.484697 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14797
W20260928 15:07:23.489280 14782 connection.cc:535] client connection to 127.0.0.1:32901 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32901): Connection reset by peer (error 104)
2026-09-28T15:07:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:23 dist-test-slave-qnz0 krb5kdc[14897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:23 dist-test-slave-qnz0 krb5kdc[14897](info): set up 2 sockets
Sep 28 15:07:23 dist-test-slave-qnz0 krb5kdc[14897](info): commencing operation
krb5kdc: starting...
W20260928 15:07:24.563455 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:24 dist-test-slave-qnz0 krb5kdc[14897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:24.692499 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:24.784336 14910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:24.784683 14910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:24.784724 14910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:24.785985 14910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:24.786056 14910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:24.786083 14910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:24.786113 14910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:24.786141 14910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:24.787724 14910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:24.788053 14910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:24.788295 14910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:24.794041 14915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:24.794222 14910 server_base.cc:1034] running on GCE node
W20260928 15:07:24.793996 14918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:24.794045 14916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:24.794580 14910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:24.794848 14910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:24.796000 14910 hybrid_clock.cc:648] HybridClock initialized: now 1790608044795991 us; error 31 us; skew 500 ppm
Sep 28 15:07:24 dist-test-slave-qnz0 krb5kdc[14897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:24.798583 14910 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:24.799052 14910 webserver.cc:460] Webserver started at http://127.0.0.1:40989/ using document root <none> and password file <none>
I20260928 15:07:24.799235 14910 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:24.799275 14910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:24.799362 14910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:24.800498 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "f839d090fc1642ce8ee09f55769be00f"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-qnz0"
server_key: "6b4d5503fd11602db8c97879c1de558d"
server_key_iv: "808003cc126ea989822350abbe2a6678"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.800809 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "f839d090fc1642ce8ee09f55769be00f"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-qnz0"
server_key: "6b4d5503fd11602db8c97879c1de558d"
server_key_iv: "808003cc126ea989822350abbe2a6678"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.802098 14910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:24.802903 14925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:24.803117 14910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:24.803220 14910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "f839d090fc1642ce8ee09f55769be00f"
format_stamp: "Formatted at 2026-09-28 15:07:24 on dist-test-slave-qnz0"
server_key: "6b4d5503fd11602db8c97879c1de558d"
server_key_iv: "808003cc126ea989822350abbe2a6678"
server_key_version: "encryptionkey@0"
I20260928 15:07:24.803295 14910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:24.806633 14910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:24.808874 14910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:24.809026 14910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:24.813347 14910 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38473
I20260928 15:07:24.813390 14978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38473 every 8 connection(s)
I20260928 15:07:24.813747 14910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:24.814811 14979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:24.817484 14979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f: Bootstrap starting.
I20260928 15:07:24.817678 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 14910
I20260928 15:07:24.817811 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:24.818044 26092 external_mini_cluster.cc:1428] Setting key 41677f29d73b4a0792e35253ebf47fa7
I20260928 15:07:24.818329 14979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:24.818715 14979 log.cc:824] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:24.819526 14979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f: No bootstrap required, opened a new log
I20260928 15:07:24.820917 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:24.818624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:55875 (local address 127.0.0.1:38473)
0928 15:07:24.818857 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:24.818867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:24.818899 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:07:24.819460 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:24.819468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:24.819495 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:24.819659 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:24.820342 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:24.820345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:24.820345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:24.820349 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:24.820453 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:24.820471 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:24.820561 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:24.820637 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 15:07:24.820689 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
I20260928 15:07:24.822062 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:24.822371 14979 raft_consensus.cc:348] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839d090fc1642ce8ee09f55769be00f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38473 } }
I20260928 15:07:24.822522 14979 raft_consensus.cc:374] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:24.822547 14979 raft_consensus.cc:729] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f839d090fc1642ce8ee09f55769be00f, State: Initialized, Role: FOLLOWER
I20260928 15:07:24.822695 14979 consensus_queue.cc:260] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839d090fc1642ce8ee09f55769be00f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38473 } }
I20260928 15:07:24.822779 14979 raft_consensus.cc:388] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:24.822803 14979 raft_consensus.cc:482] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:24.822834 14979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:24.823670 14979 raft_consensus.cc:504] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839d090fc1642ce8ee09f55769be00f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38473 } }
I20260928 15:07:24.823837 14979 leader_election.cc:302] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f839d090fc1642ce8ee09f55769be00f; no voters:
I20260928 15:07:24.824033 14979 leader_election.cc:288] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:24.824055 14989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:24.824225 14989 raft_consensus.cc:686] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [term 1 LEADER]: Becoming Leader. State: Replica: f839d090fc1642ce8ee09f55769be00f, State: Running, Role: LEADER
I20260928 15:07:24.824324 14979 sys_catalog.cc:564] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:24.824371 14989 consensus_queue.cc:237] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839d090fc1642ce8ee09f55769be00f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38473 } }
I20260928 15:07:24.824883 14989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f839d090fc1642ce8ee09f55769be00f. Latest consensus state: current_term: 1 leader_uuid: "f839d090fc1642ce8ee09f55769be00f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839d090fc1642ce8ee09f55769be00f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38473 } } }
I20260928 15:07:24.824975 14989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:24.825114 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:24.823703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:49389 (local address 127.0.0.1:38473)
0928 15:07:24.823748 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:24.823750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:24.823864 (+ 114us) server_negotiation.cc:407] Connection header received
0928 15:07:24.823961 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:24.823962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:24.823967 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:24.824016 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:24.824876 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:24.824877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:24.824878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:24.824883 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:24.824898 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:24.824902 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:24.824957 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:24.825031 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 15:07:24.825046 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 15:07:24.825104 14990 sys_catalog.cc:455] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f839d090fc1642ce8ee09f55769be00f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839d090fc1642ce8ee09f55769be00f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38473 } } }
I20260928 15:07:24.825575 14990 sys_catalog.cc:458] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:24.825814 14996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:24.826354 14996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:24.828075 14996 catalog_manager.cc:1347] Generated new cluster ID: 91202bf909064db08912bccbffbc1fec
I20260928 15:07:24.828135 14996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:24.838414 14996 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:24.838608 14996 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:24.843449 14996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f839d090fc1642ce8ee09f55769be00f: Generated new TSK 0
I20260928 15:07:24.843672 14996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:24.845577 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 14910
2026-09-28T15:07:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:24 dist-test-slave-qnz0 krb5kdc[15010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:24 dist-test-slave-qnz0 krb5kdc[15010](info): set up 2 sockets
Sep 28 15:07:24 dist-test-slave-qnz0 krb5kdc[15010](info): commencing operation
krb5kdc: starting...
W20260928 15:07:25.915702 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:26 dist-test-slave-qnz0 krb5kdc[15010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:26.051837 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:26.137414 15023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:26.137642 15023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:26.137667 15023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:26.138833 15023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:26.138876 15023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:26.138891 15023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:26.138904 15023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:26.138916 15023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:26.140353 15023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:26.140631 15023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:26.140825 15023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:26.146042 15031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:26.146201 15023 server_base.cc:1034] running on GCE node
W20260928 15:07:26.146128 15029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:26.146059 15028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:26.146574 15023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:26.146888 15023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:26.148070 15023 hybrid_clock.cc:648] HybridClock initialized: now 1790608046148047 us; error 36 us; skew 500 ppm
Sep 28 15:07:26 dist-test-slave-qnz0 krb5kdc[15010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:26.158973 15023 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:26.159615 15023 webserver.cc:460] Webserver started at http://127.0.0.1:45933/ using document root <none> and password file <none>
I20260928 15:07:26.159964 15023 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:26.160023 15023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:26.160167 15023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:26.161477 15023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "2ecd41c3ca104394b64da4a049cfd74f"
format_stamp: "Formatted at 2026-09-28 15:07:26 on dist-test-slave-qnz0"
server_key: "20aaf17cd875976ed668c37781317e61"
server_key_iv: "b9ffbc5bfe9ef51ef617ef98da11ad6f"
server_key_version: "encryptionkey@0"
I20260928 15:07:26.161883 15023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "2ecd41c3ca104394b64da4a049cfd74f"
format_stamp: "Formatted at 2026-09-28 15:07:26 on dist-test-slave-qnz0"
server_key: "20aaf17cd875976ed668c37781317e61"
server_key_iv: "b9ffbc5bfe9ef51ef617ef98da11ad6f"
server_key_version: "encryptionkey@0"
I20260928 15:07:26.163434 15023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 15:07:26.164367 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:26.164637 15023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:26.164758 15023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "2ecd41c3ca104394b64da4a049cfd74f"
format_stamp: "Formatted at 2026-09-28 15:07:26 on dist-test-slave-qnz0"
server_key: "20aaf17cd875976ed668c37781317e61"
server_key_iv: "b9ffbc5bfe9ef51ef617ef98da11ad6f"
server_key_version: "encryptionkey@0"
I20260928 15:07:26.164883 15023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:26.202901 15023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:26.205263 15023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:26.205545 15023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:26.209676 15023 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36835
I20260928 15:07:26.209709 15090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36835 every 8 connection(s)
I20260928 15:07:26.210098 15023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:26.210990 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:26.213304 15091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f: Bootstrap starting.
I20260928 15:07:26.213886 15091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:26.214169 15091 log.cc:824] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:26.214907 15091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f: No bootstrap required, opened a new log
I20260928 15:07:26.216550 15091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecd41c3ca104394b64da4a049cfd74f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 15:07:26.216689 15091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:26.216712 15091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecd41c3ca104394b64da4a049cfd74f, State: Initialized, Role: FOLLOWER
I20260928 15:07:26.216871 15091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecd41c3ca104394b64da4a049cfd74f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 15:07:26.216934 15091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:26.216956 15091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:26.217021 15091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:26.217628 15091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecd41c3ca104394b64da4a049cfd74f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 15:07:26.217715 15091 leader_election.cc:302] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ecd41c3ca104394b64da4a049cfd74f; no voters:
I20260928 15:07:26.217912 15091 leader_election.cc:288] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:26.217947 15094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:26.218140 15094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [term 1 LEADER]: Becoming Leader. State: Replica: 2ecd41c3ca104394b64da4a049cfd74f, State: Running, Role: LEADER
I20260928 15:07:26.218191 15091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:26.218268 15094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecd41c3ca104394b64da4a049cfd74f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 15:07:26.218746 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15023
I20260928 15:07:26.218875 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:26.218833 15096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ecd41c3ca104394b64da4a049cfd74f. Latest consensus state: current_term: 1 leader_uuid: "2ecd41c3ca104394b64da4a049cfd74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecd41c3ca104394b64da4a049cfd74f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } } }
I20260928 15:07:26.218917 15096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:26.219105 26092 external_mini_cluster.cc:1428] Setting key 0a80db56f25fbd44fc42e95dab1b544b
I20260928 15:07:26.219259 15095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ecd41c3ca104394b64da4a049cfd74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecd41c3ca104394b64da4a049cfd74f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } } }
I20260928 15:07:26.219413 15095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:26.219535 15103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:26.220067 15103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:26.221642 15103 catalog_manager.cc:1347] Generated new cluster ID: 1b432ebd511641e2bcf11247b094a436
I20260928 15:07:26.221695 15103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 15:07:26 dist-test-slave-qnz0 krb5kdc[15010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:07:26.231524 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.219739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:44635 (local address 127.0.0.1:36835)
0928 15:07:26.220108 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.220117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.220150 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:07:26.220265 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.220272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.220333 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.220505 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:26.221356 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.222732 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.223847 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.224511 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.224614 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.227626 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:26.227631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:26.227632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:26.227646 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:26.229727 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.230179 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.230183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.230183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.230308 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.230551 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.230552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.230553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.230885 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:26.231052 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.231159 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.231268 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":88,"threads_started":1}
I20260928 15:07:26.232796 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:26.241698 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:26.234260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:51947 (local address 127.0.0.1:36835)
0928 15:07:26.234308 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:26.234310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:26.234428 (+ 118us) server_negotiation.cc:407] Connection header received
0928 15:07:26.234455 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:26.234456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:26.234466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:26.234535 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:26.235134 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.235936 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.236920 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:26.237258 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:26.237331 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:26.238760 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:26.238762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:26.238762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:26.238769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:26.240468 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.240874 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.240875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.240875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.240935 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:26.241169 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:26.241170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:26.241170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:26.241301 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:26.241407 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:26.241536 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:07:26.241599 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 15:07:26.248521 15103 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:26.249059 15103 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:26.252380 15103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ecd41c3ca104394b64da4a049cfd74f: Generated new TSK 0
I20260928 15:07:26.252595 15103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:26.260190 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15023
W20260928 15:07:26.264438 15008 connection.cc:535] client connection to 127.0.0.1:36835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36835): Connection reset by peer (error 104)
2026-09-28T15:07:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:26 dist-test-slave-qnz0 krb5kdc[15122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:26 dist-test-slave-qnz0 krb5kdc[15122](info): set up 2 sockets
Sep 28 15:07:26 dist-test-slave-qnz0 krb5kdc[15122](info): commencing operation
krb5kdc: starting...
W20260928 15:07:27.336150 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:27 dist-test-slave-qnz0 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:27.471613 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:27.561405 15135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:27.561651 15135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:27.561677 15135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:27.562906 15135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:27.563068 15135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:27.563127 15135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:27.563143 15135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:27.563158 15135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:27.565070 15135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:27.565321 15135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:27.565505 15135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:27.570794 15141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:27.570809 15140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:27.571071 15135 server_base.cc:1034] running on GCE node
W20260928 15:07:27.570828 15143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:27.571483 15135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:27.571874 15135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:27.573055 15135 hybrid_clock.cc:648] HybridClock initialized: now 1790608047573047 us; error 40 us; skew 500 ppm
Sep 28 15:07:27 dist-test-slave-qnz0 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:27.575773 15135 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:27.576236 15135 webserver.cc:460] Webserver started at http://127.0.0.1:37117/ using document root <none> and password file <none>
I20260928 15:07:27.576527 15135 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:27.576596 15135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:27.576761 15135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:27.577900 15135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "ea858596048b4f0ab1a1c97ee226eb92"
format_stamp: "Formatted at 2026-09-28 15:07:27 on dist-test-slave-qnz0"
server_key: "109e3d921caa4a4bb168931bfbe2f8c9"
server_key_iv: "345ee3a4b3b1707788307d7c6d0fa1aa"
server_key_version: "encryptionkey@0"
I20260928 15:07:27.578276 15135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "ea858596048b4f0ab1a1c97ee226eb92"
format_stamp: "Formatted at 2026-09-28 15:07:27 on dist-test-slave-qnz0"
server_key: "109e3d921caa4a4bb168931bfbe2f8c9"
server_key_iv: "345ee3a4b3b1707788307d7c6d0fa1aa"
server_key_version: "encryptionkey@0"
I20260928 15:07:27.579667 15135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:27.580618 15150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:27.580928 15135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:27.581058 15135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "ea858596048b4f0ab1a1c97ee226eb92"
format_stamp: "Formatted at 2026-09-28 15:07:27 on dist-test-slave-qnz0"
server_key: "109e3d921caa4a4bb168931bfbe2f8c9"
server_key_iv: "345ee3a4b3b1707788307d7c6d0fa1aa"
server_key_version: "encryptionkey@0"
I20260928 15:07:27.581138 15135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:27.605185 15135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:27.607470 15135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:27.607640 15135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:27.612015 15135 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42645
I20260928 15:07:27.612052 15202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42645 every 8 connection(s)
I20260928 15:07:27.612490 15135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:27.613451 15203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:27.616266 15203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92: Bootstrap starting.
I20260928 15:07:27.617077 15203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:27.617399 15203 log.cc:824] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:27.617642 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15135
I20260928 15:07:27.617792 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:27.618036 26092 external_mini_cluster.cc:1428] Setting key 3ab417b8368060619b42b931d1c8d2e3
I20260928 15:07:27.618311 15203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92: No bootstrap required, opened a new log
I20260928 15:07:27.620970 15203 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea858596048b4f0ab1a1c97ee226eb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42645 } }
I20260928 15:07:27.621127 15203 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:27.621150 15203 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea858596048b4f0ab1a1c97ee226eb92, State: Initialized, Role: FOLLOWER
I20260928 15:07:27.621296 15203 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea858596048b4f0ab1a1c97ee226eb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42645 } }
I20260928 15:07:27.621388 15203 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:27.621531 15203 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:27.621593 15203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:27.622205 15203 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea858596048b4f0ab1a1c97ee226eb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42645 } }
I20260928 15:07:27.622325 15203 leader_election.cc:302] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea858596048b4f0ab1a1c97ee226eb92; no voters:
I20260928 15:07:27.622581 15203 leader_election.cc:288] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:27.622618 15208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:27.622805 15208 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [term 1 LEADER]: Becoming Leader. State: Replica: ea858596048b4f0ab1a1c97ee226eb92, State: Running, Role: LEADER
I20260928 15:07:27.622949 15208 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea858596048b4f0ab1a1c97ee226eb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42645 } }
I20260928 15:07:27.622954 15203 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:27.623395 15209 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea858596048b4f0ab1a1c97ee226eb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea858596048b4f0ab1a1c97ee226eb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42645 } } }
I20260928 15:07:27.623399 15210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea858596048b4f0ab1a1c97ee226eb92. Latest consensus state: current_term: 1 leader_uuid: "ea858596048b4f0ab1a1c97ee226eb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea858596048b4f0ab1a1c97ee226eb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42645 } } }
I20260928 15:07:27.623559 15209 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:27.623566 15210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:27.624363 15213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:27.624931 15213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:27.626595 15213 catalog_manager.cc:1347] Generated new cluster ID: 748e424a784b48d4a0958d4d450b91e4
I20260928 15:07:27.626657 15213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:27.626863 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:27.618818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:41931 (local address 127.0.0.1:42645)
0928 15:07:27.619038 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:27.619048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:27.619081 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:07:27.621114 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:27.621140 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:27.621221 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:27.621388 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:27.621623 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.622708 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.624149 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.624942 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.625067 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:27.626095 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:27.626099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:27.626099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:27.626334 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:27.626367 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:27.626379 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:27.626483 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:27.626578 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 15:07:27.626610 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260928 15:07:27.627985 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:27.635514 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:27.629467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:47849 (local address 127.0.0.1:42645)
0928 15:07:27.629509 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:27.629511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:27.632629 (+ 3118us) server_negotiation.cc:407] Connection header received
0928 15:07:27.632719 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:27.632721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:27.632735 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:27.632797 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:27.632969 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.633490 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.634213 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:27.634435 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:27.634510 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:27.635225 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:27.635226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:27.635227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:27.635232 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:27.635248 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:27.635253 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:27.635332 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:27.635391 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 15:07:27.635402 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 15:07:27.639122 15213 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:27.639945 15213 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:27.650628 15213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea858596048b4f0ab1a1c97ee226eb92: Generated new TSK 0
I20260928 15:07:27.650894 15213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:27.660550 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15135
W20260928 15:07:27.665035 15120 connection.cc:535] client connection to 127.0.0.1:42645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42645): Connection reset by peer (error 104)
2026-09-28T15:07:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:27 dist-test-slave-qnz0 krb5kdc[15234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:27 dist-test-slave-qnz0 krb5kdc[15234](info): set up 2 sockets
Sep 28 15:07:27 dist-test-slave-qnz0 krb5kdc[15234](info): commencing operation
krb5kdc: starting...
W20260928 15:07:28.731346 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:28 dist-test-slave-qnz0 krb5kdc[15234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:07:28.866106 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:28.955130 15247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:28.955431 15247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:28.955477 15247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:28.956748 15247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:28.956825 15247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:28.956847 15247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:28.956867 15247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:28.956894 15247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:28.958508 15247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:28.958796 15247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:28.959039 15247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:07:28.964454 15247 server_base.cc:1034] running on GCE node
W20260928 15:07:28.964378 15255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:28.964519 15252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:28.964378 15253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:28.964860 15247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:28.965179 15247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:28.966331 15247 hybrid_clock.cc:648] HybridClock initialized: now 1790608048966320 us; error 30 us; skew 500 ppm
Sep 28 15:07:28 dist-test-slave-qnz0 krb5kdc[15234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:28.969127 15247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:07:28.969659 15247 webserver.cc:460] Webserver started at http://127.0.0.1:45193/ using document root <none> and password file <none>
I20260928 15:07:28.969913 15247 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:28.969991 15247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:28.970135 15247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:28.971258 15247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "35798c7816134397afd6dd1b222a9a5f"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-qnz0"
server_key: "99df55b4365e3907748feb16305fa9ba"
server_key_iv: "3acedfde205052e7b28dfa15131bc163"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.971627 15247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "35798c7816134397afd6dd1b222a9a5f"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-qnz0"
server_key: "99df55b4365e3907748feb16305fa9ba"
server_key_iv: "3acedfde205052e7b28dfa15131bc163"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.973151 15247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:28.973995 15262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:28.974452 15247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:28.974628 15247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "35798c7816134397afd6dd1b222a9a5f"
format_stamp: "Formatted at 2026-09-28 15:07:28 on dist-test-slave-qnz0"
server_key: "99df55b4365e3907748feb16305fa9ba"
server_key_iv: "3acedfde205052e7b28dfa15131bc163"
server_key_version: "encryptionkey@0"
I20260928 15:07:28.974722 15247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:28.978631 15247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:28.981385 15247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:28.981508 15247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:28.985754 15247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40503
I20260928 15:07:28.985837 15314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40503 every 8 connection(s)
I20260928 15:07:28.986156 15247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:28.987268 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:28.989702 15315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f: Bootstrap starting.
I20260928 15:07:28.990317 15315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:28.990607 15315 log.cc:824] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:28.991406 15315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f: No bootstrap required, opened a new log
I20260928 15:07:28.991667 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15247
I20260928 15:07:28.991792 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:28.991999 26092 external_mini_cluster.cc:1428] Setting key b3f57f9e1c74132d5ea5c13c1a758390
I20260928 15:07:28.993767 15315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35798c7816134397afd6dd1b222a9a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260928 15:07:28.993952 15315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:28.994032 15315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35798c7816134397afd6dd1b222a9a5f, State: Initialized, Role: FOLLOWER
I20260928 15:07:28.994194 15315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35798c7816134397afd6dd1b222a9a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260928 15:07:28.994330 15315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:28.994367 15315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:28.994417 15315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:28.995326 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:28.992583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:48541 (local address 127.0.0.1:40503)
0928 15:07:28.992770 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:28.992777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:28.992807 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:07:28.993772 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:28.993780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:28.993830 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:28.994017 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:28.994796 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:28.994800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:28.994800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:28.994806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:28.994853 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:28.994872 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:28.994966 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:28.995031 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 15:07:28.995085 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260928 15:07:28.995632 15315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35798c7816134397afd6dd1b222a9a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260928 15:07:28.995750 15315 leader_election.cc:302] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35798c7816134397afd6dd1b222a9a5f; no voters:
I20260928 15:07:28.995939 15315 leader_election.cc:288] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:28.996037 15320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:28.996206 15315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:28.996256 15320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [term 1 LEADER]: Becoming Leader. State: Replica: 35798c7816134397afd6dd1b222a9a5f, State: Running, Role: LEADER
I20260928 15:07:28.996344 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:28.996533 15320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35798c7816134397afd6dd1b222a9a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260928 15:07:28.996940 15320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35798c7816134397afd6dd1b222a9a5f. Latest consensus state: current_term: 1 leader_uuid: "35798c7816134397afd6dd1b222a9a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35798c7816134397afd6dd1b222a9a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } } }
I20260928 15:07:28.996954 15321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35798c7816134397afd6dd1b222a9a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35798c7816134397afd6dd1b222a9a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } } }
I20260928 15:07:28.997094 15320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:28.997164 15321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:28.997347 15330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:28.997975 15330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:28.999387 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:28.998043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:48171 (local address 127.0.0.1:40503)
0928 15:07:28.998080 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:28.998082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:28.998240 (+ 158us) server_negotiation.cc:407] Connection header received
0928 15:07:28.998337 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:28.998338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:28.998343 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:28.998383 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:28.999130 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:28.999132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:28.999132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:28.999136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:28.999157 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:28.999160 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:28.999214 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:28.999283 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 15:07:28.999299 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 15:07:28.999895 15330 catalog_manager.cc:1347] Generated new cluster ID: 3ebc53f2d7a544b39a26ccb32496b419
I20260928 15:07:28.999956 15330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:29.007932 15330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:29.008154 15330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:29.019575 15330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35798c7816134397afd6dd1b222a9a5f: Generated new TSK 0
I20260928 15:07:29.019866 15330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:29.022985 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15247
2026-09-28T15:07:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:29 dist-test-slave-qnz0 krb5kdc[15346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:29 dist-test-slave-qnz0 krb5kdc[15346](info): set up 2 sockets
Sep 28 15:07:29 dist-test-slave-qnz0 krb5kdc[15346](info): commencing operation
krb5kdc: starting...
W20260928 15:07:30.083369 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:30 dist-test-slave-qnz0 krb5kdc[15346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:30.220815 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37549
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:37549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:30.309744 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:30.310034 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:30.310075 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:30.311338 15359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:30.311416 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:30.311441 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:30.311465 15359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:30.311493 15359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:30.313098 15359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:37549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:30.313418 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:30.313670 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:30.318912 15365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:30.318922 15364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:30.318909 15367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:30.319175 15359 server_base.cc:1034] running on GCE node
I20260928 15:07:30.319447 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:30.319742 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:30.320888 15359 hybrid_clock.cc:648] HybridClock initialized: now 1790608050320856 us; error 62 us; skew 500 ppm
Sep 28 15:07:30 dist-test-slave-qnz0 krb5kdc[15346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608050, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:30.323709 15359 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:30.324281 15359 webserver.cc:460] Webserver started at http://127.25.123.62:34543/ using document root <none> and password file <none>
I20260928 15:07:30.324600 15359 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:30.324678 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:30.324844 15359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:30.326032 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "3d3978bb2a4e41a9a96f124199f43e80"
format_stamp: "Formatted at 2026-09-28 15:07:30 on dist-test-slave-qnz0"
server_key: "ae9e0925a7637e097cf664a4b4c4f017"
server_key_iv: "a7f80bce8a4060756cbad79e0c2eb3f6"
server_key_version: "encryptionkey@0"
I20260928 15:07:30.326439 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "3d3978bb2a4e41a9a96f124199f43e80"
format_stamp: "Formatted at 2026-09-28 15:07:30 on dist-test-slave-qnz0"
server_key: "ae9e0925a7637e097cf664a4b4c4f017"
server_key_iv: "a7f80bce8a4060756cbad79e0c2eb3f6"
server_key_version: "encryptionkey@0"
I20260928 15:07:30.327946 15359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:30.328954 15374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:30.329188 15359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:07:30.329339 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "3d3978bb2a4e41a9a96f124199f43e80"
format_stamp: "Formatted at 2026-09-28 15:07:30 on dist-test-slave-qnz0"
server_key: "ae9e0925a7637e097cf664a4b4c4f017"
server_key_iv: "a7f80bce8a4060756cbad79e0c2eb3f6"
server_key_version: "encryptionkey@0"
I20260928 15:07:30.329424 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:30.348251 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:30.350561 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:30.350754 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:30.355188 15359 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:37549
I20260928 15:07:30.355225 15426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:37549 every 8 connection(s)
I20260928 15:07:30.355588 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:30.356527 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15359
I20260928 15:07:30.356663 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:30.356650 15427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:30.356878 26092 external_mini_cluster.cc:1428] Setting key 84b4230f8d49542356dc4e8e9eeeda3d
I20260928 15:07:30.360067 15427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80: Bootstrap starting.
I20260928 15:07:30.360842 15427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:30.361155 15427 log.cc:824] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:30.362025 15427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80: No bootstrap required, opened a new log
Sep 28 15:07:30 dist-test-slave-qnz0 krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:30.364908 15427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3978bb2a4e41a9a96f124199f43e80" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37549 } }
I20260928 15:07:30.365096 15427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:30.365123 15427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3978bb2a4e41a9a96f124199f43e80, State: Initialized, Role: FOLLOWER
I20260928 15:07:30.365263 15427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3978bb2a4e41a9a96f124199f43e80" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37549 } }
I20260928 15:07:30.365339 15427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:30.365362 15427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:30.365435 15427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:30.366197 15427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3978bb2a4e41a9a96f124199f43e80" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37549 } }
I20260928 15:07:30.366335 15427 leader_election.cc:302] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d3978bb2a4e41a9a96f124199f43e80; no voters:
I20260928 15:07:30.366566 15432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:30.366532 15427 leader_election.cc:288] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:30.366731 15432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [term 1 LEADER]: Becoming Leader. State: Replica: 3d3978bb2a4e41a9a96f124199f43e80, State: Running, Role: LEADER
I20260928 15:07:30.366899 15427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:30.366871 15432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3978bb2a4e41a9a96f124199f43e80" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37549 } }
I20260928 15:07:30.368855 15434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d3978bb2a4e41a9a96f124199f43e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3978bb2a4e41a9a96f124199f43e80" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37549 } } }
I20260928 15:07:30.368949 15434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:30.369202 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:30.357462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51823 (local address 127.25.123.62:37549)
0928 15:07:30.357620 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:30.357627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:30.357651 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:07:30.358221 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:30.358239 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:30.358287 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:30.358416 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:30.359266 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:30.360271 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:30.361376 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:30.361920 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:30.362012 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:30.365038 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:30.365042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:30.365042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:30.365053 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:30.367260 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:30.367741 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:30.367744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:30.367745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:30.367836 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:30.368126 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:30.368128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:30.368128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:30.368502 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:30.368677 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:30.368833 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 15:07:30.368937 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260928 15:07:30.369518 15433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d3978bb2a4e41a9a96f124199f43e80. Latest consensus state: current_term: 1 leader_uuid: "3d3978bb2a4e41a9a96f124199f43e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3978bb2a4e41a9a96f124199f43e80" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 37549 } } }
I20260928 15:07:30.369639 15446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:30.369979 15433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:30.370358 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:30.370497 15446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:30.373149 15446 catalog_manager.cc:1347] Generated new cluster ID: 1a0271f3ca534e6fb8e8f3993d27268f
I20260928 15:07:30.373205 15446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:30.379245 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:30.372080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41719 (local address 127.25.123.62:37549)
0928 15:07:30.372122 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:30.372124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:30.372218 (+ 94us) server_negotiation.cc:407] Connection header received
0928 15:07:30.372280 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:30.372281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:30.372291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:30.372352 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:30.373193 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:30.373749 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:30.374560 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:30.374825 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:30.374920 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:30.376254 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:30.376255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:30.376256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:30.376263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:30.377873 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:30.378298 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:30.378300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:30.378300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:30.378364 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:30.378632 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:30.378633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:30.378634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:30.378755 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:30.378881 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:30.379039 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 15:07:30.379110 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:07:30.412662 15446 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:30.413439 15446 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:30.424369 15446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d3978bb2a4e41a9a96f124199f43e80: Generated new TSK 0
I20260928 15:07:30.424631 15446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:30.429806 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15359
W20260928 15:07:30.434425 15344 connection.cc:535] client connection to 127.25.123.62:37549 recv error: Network error: failed to read from TLS socket (remote: 127.25.123.62:37549): Connection reset by peer (error 104)
2026-09-28T15:07:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:30 dist-test-slave-qnz0 krb5kdc[15458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:30 dist-test-slave-qnz0 krb5kdc[15458](info): set up 2 sockets
Sep 28 15:07:30 dist-test-slave-qnz0 krb5kdc[15458](info): commencing operation
krb5kdc: starting...
W20260928 15:07:31.511307 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:31 dist-test-slave-qnz0 krb5kdc[15458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:31.636766 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:42765
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:31.722460 15471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:31.722677 15471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:31.722702 15471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:31.723870 15471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:31.723917 15471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:31.723932 15471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:31.723945 15471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:31.723958 15471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:31.725556 15471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:42765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:31.725827 15471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:31.726068 15471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:31.732520 15477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:31.732635 15476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:31.732525 15479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:31.732820 15471 server_base.cc:1034] running on GCE node
I20260928 15:07:31.733109 15471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:31.733440 15471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:31.734612 15471 hybrid_clock.cc:648] HybridClock initialized: now 1790608051734559 us; error 71 us; skew 500 ppm
Sep 28 15:07:31 dist-test-slave-qnz0 krb5kdc[15458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608051, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:31.743815 15471 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:31.744306 15471 webserver.cc:460] Webserver started at http://127.25.123.62:45993/ using document root <none> and password file <none>
I20260928 15:07:31.744597 15471 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:31.744650 15471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:31.744781 15471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:31.745905 15471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "6f3fb2750a704df49d62e619027f4201"
format_stamp: "Formatted at 2026-09-28 15:07:31 on dist-test-slave-qnz0"
server_key: "885820ff081e2fe0d55952502d764d4a"
server_key_iv: "7748f87ac74e354b368e1d72d4a65fd1"
server_key_version: "encryptionkey@0"
I20260928 15:07:31.746279 15471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "6f3fb2750a704df49d62e619027f4201"
format_stamp: "Formatted at 2026-09-28 15:07:31 on dist-test-slave-qnz0"
server_key: "885820ff081e2fe0d55952502d764d4a"
server_key_iv: "7748f87ac74e354b368e1d72d4a65fd1"
server_key_version: "encryptionkey@0"
I20260928 15:07:31.747748 15471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:31.748694 15486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:31.748970 15471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:31.749063 15471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "6f3fb2750a704df49d62e619027f4201"
format_stamp: "Formatted at 2026-09-28 15:07:31 on dist-test-slave-qnz0"
server_key: "885820ff081e2fe0d55952502d764d4a"
server_key_iv: "7748f87ac74e354b368e1d72d4a65fd1"
server_key_version: "encryptionkey@0"
I20260928 15:07:31.749161 15471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:31.756173 15471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:31.758472 15471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:31.758641 15471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:31.763115 15471 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:42765
I20260928 15:07:31.763751 15471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:31.763844 15538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:42765 every 8 connection(s)
I20260928 15:07:31.764663 15539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:31.766955 15539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201: Bootstrap starting.
I20260928 15:07:31.767509 15539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:31.767750 15539 log.cc:824] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:31.768719 15539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201: No bootstrap required, opened a new log
I20260928 15:07:31.770283 15539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3fb2750a704df49d62e619027f4201" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 42765 } }
I20260928 15:07:31.770437 15539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:31.770460 15539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3fb2750a704df49d62e619027f4201, State: Initialized, Role: FOLLOWER
I20260928 15:07:31.770606 15539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3fb2750a704df49d62e619027f4201" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 42765 } }
I20260928 15:07:31.770692 15539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:31.770715 15539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:31.770745 15539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:31.771607 15539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3fb2750a704df49d62e619027f4201" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 42765 } }
I20260928 15:07:31.771700 15539 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3fb2750a704df49d62e619027f4201; no voters:
I20260928 15:07:31.771843 15539 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:31.771930 15542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:31.772159 15542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [term 1 LEADER]: Becoming Leader. State: Replica: 6f3fb2750a704df49d62e619027f4201, State: Running, Role: LEADER
I20260928 15:07:31.772270 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15471
I20260928 15:07:31.772321 15542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3fb2750a704df49d62e619027f4201" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 42765 } }
I20260928 15:07:31.772451 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:31.772641 26092 external_mini_cluster.cc:1428] Setting key a2720ad5223405caff73787a075c6760
I20260928 15:07:31.772828 15544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3fb2750a704df49d62e619027f4201. Latest consensus state: current_term: 1 leader_uuid: "6f3fb2750a704df49d62e619027f4201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3fb2750a704df49d62e619027f4201" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 42765 } } }
I20260928 15:07:31.772944 15544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:31.773138 15543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f3fb2750a704df49d62e619027f4201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3fb2750a704df49d62e619027f4201" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 42765 } } }
I20260928 15:07:31.772871 15539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:31.773216 15543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:31.773443 15546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:31.774273 15546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:31.776526 15546 catalog_manager.cc:1347] Generated new cluster ID: f7420ed97c2849f6bcaac00b5ca62653
I20260928 15:07:31.776585 15546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:31.779331 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:31.773182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33739 (local address 127.25.123.62:42765)
0928 15:07:31.773340 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:31.773349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:31.773385 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:07:31.773563 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:31.773571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:31.773635 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:31.773788 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:31.774012 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:31.776185 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:31.777302 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:31.777708 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:31.777865 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:31.778648 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:31.778652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:31.778652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:31.778863 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:31.778897 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:31.778909 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:31.779011 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:31.779094 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 15:07:31.779124 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 15:07:31.780931 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:31.785894 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:31.782651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53377 (local address 127.25.123.62:42765)
0928 15:07:31.782715 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:31.782716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:31.782844 (+ 128us) server_negotiation.cc:407] Connection header received
0928 15:07:31.782924 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:31.782925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:31.782935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:31.782995 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:31.783141 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:31.783659 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:31.784590 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:31.784814 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:31.784893 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:31.785626 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:31.785628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:31.785628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:31.785633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:31.785649 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:31.785653 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:31.785708 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:31.785770 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 15:07:31.785782 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:07:31.792285 15546 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:31.792877 15546 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:31.806761 15546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f3fb2750a704df49d62e619027f4201: Generated new TSK 0
I20260928 15:07:31.807009 15546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:31.818558 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15471
W20260928 15:07:31.823450 15456 connection.cc:535] client connection to 127.25.123.62:42765 recv error: Network error: failed to read from TLS socket (remote: 127.25.123.62:42765): Connection reset by peer (error 104)
2026-09-28T15:07:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:31 dist-test-slave-qnz0 krb5kdc[15570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:31 dist-test-slave-qnz0 krb5kdc[15570](info): set up 2 sockets
Sep 28 15:07:31 dist-test-slave-qnz0 krb5kdc[15570](info): commencing operation
krb5kdc: starting...
W20260928 15:07:32.895746 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:32 dist-test-slave-qnz0 krb5kdc[15570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:33.033362 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:32931
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:32931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:33.118181 15583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:33.118408 15583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:33.118432 15583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:33.119628 15583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:33.119680 15583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:33.119694 15583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:33.119709 15583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:33.119720 15583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:33.121376 15583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:32931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:32931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:33.121654 15583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:33.121896 15583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:33.127146 15591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:33.127137 15589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:33.127137 15588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:33.127240 15583 server_base.cc:1034] running on GCE node
I20260928 15:07:33.127683 15583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:33.128054 15583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:33.129292 15583 hybrid_clock.cc:648] HybridClock initialized: now 1790608053129240 us; error 66 us; skew 500 ppm
Sep 28 15:07:33 dist-test-slave-qnz0 krb5kdc[15570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608053, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:33.132038 15583 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:33.132536 15583 webserver.cc:460] Webserver started at http://127.25.123.62:35861/ using document root <none> and password file <none>
I20260928 15:07:33.132757 15583 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:33.132800 15583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:33.132894 15583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:33.133970 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "be768328a56f49e389a1c6d585d035bb"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-qnz0"
server_key: "52f4b6b4003ae20850e227e21ba51875"
server_key_iv: "c83f068b72b1967761cc1f2a23801bb8"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.134269 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "be768328a56f49e389a1c6d585d035bb"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-qnz0"
server_key: "52f4b6b4003ae20850e227e21ba51875"
server_key_iv: "c83f068b72b1967761cc1f2a23801bb8"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.135588 15583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:33.136432 15598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:33.136637 15583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:33.136713 15583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "be768328a56f49e389a1c6d585d035bb"
format_stamp: "Formatted at 2026-09-28 15:07:33 on dist-test-slave-qnz0"
server_key: "52f4b6b4003ae20850e227e21ba51875"
server_key_iv: "c83f068b72b1967761cc1f2a23801bb8"
server_key_version: "encryptionkey@0"
I20260928 15:07:33.136811 15583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:33.140473 15583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:33.142928 15583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:33.143085 15583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:33.147990 15583 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:32931
I20260928 15:07:33.148021 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:32931 every 8 connection(s)
I20260928 15:07:33.148483 15583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:33.148941 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15583
I20260928 15:07:33.149063 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:33.149253 26092 external_mini_cluster.cc:1428] Setting key 78de9c9e2a10c8227ac80dc8318f325f
I20260928 15:07:33.149767 15651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:33.151975 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.149800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43869 (local address 127.25.123.62:32931)
0928 15:07:33.149979 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.149993 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.150044 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:07:33.150482 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.150488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.150511 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.150657 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:33.151443 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:33.151449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:33.151449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:33.151458 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:33.151506 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:33.151517 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:33.151707 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.151768 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.151811 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260928 15:07:33.153167 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:33.153254 15651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb: Bootstrap starting.
I20260928 15:07:33.153877 15651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:33.154181 15651 log.cc:824] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:33.155103 15651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb: No bootstrap required, opened a new log
I20260928 15:07:33.156584 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:33.155055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57657 (local address 127.25.123.62:32931)
0928 15:07:33.155102 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:33.155104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:33.155232 (+ 128us) server_negotiation.cc:407] Connection header received
0928 15:07:33.155323 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:33.155324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:33.155329 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:33.155403 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:33.156107 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:33.156109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:33.156109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:33.156114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:33.156129 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:33.156134 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:33.156206 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:33.156322 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 15:07:33.156337 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:07:33.157716 15651 raft_consensus.cc:348] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be768328a56f49e389a1c6d585d035bb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32931 } }
I20260928 15:07:33.157893 15651 raft_consensus.cc:374] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:33.157936 15651 raft_consensus.cc:729] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be768328a56f49e389a1c6d585d035bb, State: Initialized, Role: FOLLOWER
I20260928 15:07:33.158083 15651 consensus_queue.cc:260] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be768328a56f49e389a1c6d585d035bb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32931 } }
I20260928 15:07:33.158191 15651 raft_consensus.cc:388] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:33.158238 15651 raft_consensus.cc:482] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:33.158291 15651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:33.158900 15651 raft_consensus.cc:504] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be768328a56f49e389a1c6d585d035bb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32931 } }
I20260928 15:07:33.159024 15651 leader_election.cc:302] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be768328a56f49e389a1c6d585d035bb; no voters:
I20260928 15:07:33.159247 15651 leader_election.cc:288] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:33.159365 15661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:33.159502 15661 raft_consensus.cc:686] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [term 1 LEADER]: Becoming Leader. State: Replica: be768328a56f49e389a1c6d585d035bb, State: Running, Role: LEADER
I20260928 15:07:33.159584 15651 sys_catalog.cc:564] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:33.159577 15661 consensus_queue.cc:237] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be768328a56f49e389a1c6d585d035bb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32931 } }
I20260928 15:07:33.160094 15663 sys_catalog.cc:455] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be768328a56f49e389a1c6d585d035bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be768328a56f49e389a1c6d585d035bb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32931 } } }
I20260928 15:07:33.160104 15662 sys_catalog.cc:455] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader be768328a56f49e389a1c6d585d035bb. Latest consensus state: current_term: 1 leader_uuid: "be768328a56f49e389a1c6d585d035bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be768328a56f49e389a1c6d585d035bb" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32931 } } }
I20260928 15:07:33.160297 15662 sys_catalog.cc:458] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:33.160674 15663 sys_catalog.cc:458] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:33.160681 15670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:33.161494 15670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:33.163153 15670 catalog_manager.cc:1347] Generated new cluster ID: 58499bb47d6546278ecf75d8abb208fe
I20260928 15:07:33.163226 15670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:33.184676 15670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:33.184900 15670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:33.191882 15670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be768328a56f49e389a1c6d585d035bb: Generated new TSK 0
I20260928 15:07:33.192129 15670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:33.200268 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15583
2026-09-28T15:07:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:33 dist-test-slave-qnz0 krb5kdc[15682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:33 dist-test-slave-qnz0 krb5kdc[15682](info): set up 2 sockets
Sep 28 15:07:33 dist-test-slave-qnz0 krb5kdc[15682](info): commencing operation
krb5kdc: starting...
W20260928 15:07:34.312263 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:34 dist-test-slave-qnz0 krb5kdc[15682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:34.444016 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38845
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:38845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:34.527410 15695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:34.527678 15695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:34.527716 15695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:34.528930 15695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:34.529004 15695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:34.529030 15695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:34.529050 15695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:34.529081 15695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:34.530606 15695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:38845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:38845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:34.530905 15695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:34.531144 15695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:34.536195 15700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:34.536214 15701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:34.536197 15703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:34.536695 15695 server_base.cc:1034] running on GCE node
I20260928 15:07:34.537014 15695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:34.537314 15695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:34.538488 15695 hybrid_clock.cc:648] HybridClock initialized: now 1790608054538465 us; error 32 us; skew 500 ppm
Sep 28 15:07:34 dist-test-slave-qnz0 krb5kdc[15682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608054, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:34.541626 15695 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:34.542174 15695 webserver.cc:460] Webserver started at http://127.25.123.62:39579/ using document root <none> and password file <none>
I20260928 15:07:34.542428 15695 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:34.542498 15695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:34.542654 15695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:34.543764 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5"
format_stamp: "Formatted at 2026-09-28 15:07:34 on dist-test-slave-qnz0"
server_key: "f4d199feafae3c84849e2f8606012fbf"
server_key_iv: "620f280fc06c42e4bcb5db3867b743f7"
server_key_version: "encryptionkey@0"
I20260928 15:07:34.544167 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5"
format_stamp: "Formatted at 2026-09-28 15:07:34 on dist-test-slave-qnz0"
server_key: "f4d199feafae3c84849e2f8606012fbf"
server_key_iv: "620f280fc06c42e4bcb5db3867b743f7"
server_key_version: "encryptionkey@0"
I20260928 15:07:34.545732 15695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:34.546716 15710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:34.547015 15695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:34.547147 15695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5"
format_stamp: "Formatted at 2026-09-28 15:07:34 on dist-test-slave-qnz0"
server_key: "f4d199feafae3c84849e2f8606012fbf"
server_key_iv: "620f280fc06c42e4bcb5db3867b743f7"
server_key_version: "encryptionkey@0"
I20260928 15:07:34.547242 15695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:34.564003 15695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:34.566682 15695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:34.566902 15695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:34.571802 15695 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:38845
I20260928 15:07:34.571842 15762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:38845 every 8 connection(s)
I20260928 15:07:34.572270 15695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:34.573268 15763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:34.575734 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5: Bootstrap starting.
I20260928 15:07:34.576483 15763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:34.576735 15763 log.cc:824] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:34.577513 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5: No bootstrap required, opened a new log
I20260928 15:07:34.579089 15763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38845 } }
I20260928 15:07:34.579234 15763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:34.579257 15763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f9c658d7fd4f0aa6a7eaa316a00fd5, State: Initialized, Role: FOLLOWER
I20260928 15:07:34.579372 15763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38845 } }
I20260928 15:07:34.579511 15763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:34.579568 15763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:34.579619 15763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:34.579648 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15695
I20260928 15:07:34.579770 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:34.579973 26092 external_mini_cluster.cc:1428] Setting key defbb3d4858416aeaeb405ac2c2b0595
I20260928 15:07:34.580891 15763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38845 } }
I20260928 15:07:34.581023 15763 leader_election.cc:302] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f9c658d7fd4f0aa6a7eaa316a00fd5; no voters:
I20260928 15:07:34.581226 15763 leader_election.cc:288] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:34.581301 15768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:34.581498 15768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [term 1 LEADER]: Becoming Leader. State: Replica: 84f9c658d7fd4f0aa6a7eaa316a00fd5, State: Running, Role: LEADER
I20260928 15:07:34.581523 15763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:34.581637 15768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38845 } }
I20260928 15:07:34.583292 15769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38845 } } }
I20260928 15:07:34.583468 15770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f9c658d7fd4f0aa6a7eaa316a00fd5. Latest consensus state: current_term: 1 leader_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f9c658d7fd4f0aa6a7eaa316a00fd5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 38845 } } }
W20260928 15:07:34.583530 15783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 15:07:34.583777 15769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:34.583784 15770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5 [sys.catalog]: This master's current role is: LEADER
W20260928 15:07:34.583794 15783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 15:07:34 dist-test-slave-qnz0 krb5kdc[15682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:34.591538 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:34.580553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60385 (local address 127.25.123.62:38845)
0928 15:07:34.580742 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:34.580752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:34.580813 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:07:34.580912 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:34.580918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:34.580963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:34.581118 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:34.581890 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.583259 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.584301 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.584831 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.584956 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:34.587711 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:34.587716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:34.587716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:34.587736 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:34.589750 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.590187 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.590191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.590191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.590317 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.590573 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.590574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.590574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.590846 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:34.591033 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:34.591158 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 15:07:34.591327 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260928 15:07:34.591571 15785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:34.591794 15785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:34.592697 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:34.593819 15785 catalog_manager.cc:1347] Generated new cluster ID: 9477b3b56efb4634a76c87c1f933cb19
I20260928 15:07:34.593879 15785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:34.602893 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:34.594280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49419 (local address 127.25.123.62:38845)
0928 15:07:34.594326 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:34.594328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:34.594519 (+ 191us) server_negotiation.cc:407] Connection header received
0928 15:07:34.594666 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:34.594667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:34.594680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:34.594743 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:34.595429 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.596180 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.597293 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:34.597592 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:34.597653 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:34.599380 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:34.599382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:34.599382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:34.599390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:34.601133 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.601821 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.601822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.601822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.601879 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:34.602218 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:34.602219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:34.602219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:34.602349 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:34.602447 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:34.602672 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 15:07:34.602783 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:07:34.607761 15785 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:34.608377 15785 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:34.619558 15785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84f9c658d7fd4f0aa6a7eaa316a00fd5: Generated new TSK 0
I20260928 15:07:34.619801 15785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:34.624647 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15695
2026-09-28T15:07:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:34 dist-test-slave-qnz0 krb5kdc[15794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:34 dist-test-slave-qnz0 krb5kdc[15794](info): set up 2 sockets
Sep 28 15:07:34 dist-test-slave-qnz0 krb5kdc[15794](info): commencing operation
krb5kdc: starting...
W20260928 15:07:35.715389 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:35 dist-test-slave-qnz0 krb5kdc[15794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:35.844384 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:43871
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:40997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:43871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:35.930431 15807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:35.930711 15807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:35.930750 15807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:35.931978 15807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:35.932056 15807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:35.932080 15807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:35.932101 15807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:35.932133 15807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:35.934051 15807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:40997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:43871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:35.934389 15807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:35.934646 15807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:35.940730 15812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:35.940752 15815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:35.940765 15807 server_base.cc:1034] running on GCE node
W20260928 15:07:35.940730 15813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:35.941246 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:35.941587 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:35.942765 15807 hybrid_clock.cc:648] HybridClock initialized: now 1790608055942704 us; error 102 us; skew 500 ppm
Sep 28 15:07:35 dist-test-slave-qnz0 krb5kdc[15794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608055, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:35.946031 15807 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:35.946699 15807 webserver.cc:460] Webserver started at http://127.25.123.62:36653/ using document root <none> and password file <none>
I20260928 15:07:35.947027 15807 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:35.947278 15807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:35.947505 15807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:35.948783 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "fc2cd1fda310466782c13f6d6e6728e5"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-qnz0"
server_key: "6f122885d06c302264c486299a8997d7"
server_key_iv: "35afcf46f02a4aef6d15514af9f43021"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.949323 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "fc2cd1fda310466782c13f6d6e6728e5"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-qnz0"
server_key: "6f122885d06c302264c486299a8997d7"
server_key_iv: "35afcf46f02a4aef6d15514af9f43021"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.951061 15807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:35.952126 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:35.952566 15807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:35.952685 15807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "fc2cd1fda310466782c13f6d6e6728e5"
format_stamp: "Formatted at 2026-09-28 15:07:35 on dist-test-slave-qnz0"
server_key: "6f122885d06c302264c486299a8997d7"
server_key_iv: "35afcf46f02a4aef6d15514af9f43021"
server_key_version: "encryptionkey@0"
I20260928 15:07:35.952785 15807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:35.992590 15807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:35.995329 15807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:35.995589 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:36.000707 15807 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:43871
I20260928 15:07:36.000795 15874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:43871 every 8 connection(s)
I20260928 15:07:36.001178 15807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:36.002175 15875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:36.004704 15875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5: Bootstrap starting.
I20260928 15:07:36.005294 15875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:36.005535 15875 log.cc:824] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:36.006367 15875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5: No bootstrap required, opened a new log
I20260928 15:07:36.008039 15875 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2cd1fda310466782c13f6d6e6728e5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43871 } }
I20260928 15:07:36.008203 15875 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:36.008227 15875 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2cd1fda310466782c13f6d6e6728e5, State: Initialized, Role: FOLLOWER
I20260928 15:07:36.008348 15875 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2cd1fda310466782c13f6d6e6728e5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43871 } }
I20260928 15:07:36.008473 15875 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:36.008517 15875 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:36.008551 15875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:36.009702 15875 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2cd1fda310466782c13f6d6e6728e5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43871 } }
I20260928 15:07:36.009827 15875 leader_election.cc:302] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc2cd1fda310466782c13f6d6e6728e5; no voters:
I20260928 15:07:36.009994 15875 leader_election.cc:288] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:36.010099 15878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:36.010275 15878 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [term 1 LEADER]: Becoming Leader. State: Replica: fc2cd1fda310466782c13f6d6e6728e5, State: Running, Role: LEADER
I20260928 15:07:36.010342 15875 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:36.010432 15878 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2cd1fda310466782c13f6d6e6728e5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43871 } }
I20260928 15:07:36.010733 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15807
I20260928 15:07:36.010859 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:36.011083 26092 external_mini_cluster.cc:1428] Setting key 453802affa461a084eeeac03b0a3bdfd
I20260928 15:07:36.011705 15880 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc2cd1fda310466782c13f6d6e6728e5. Latest consensus state: current_term: 1 leader_uuid: "fc2cd1fda310466782c13f6d6e6728e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2cd1fda310466782c13f6d6e6728e5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43871 } } }
I20260928 15:07:36.011814 15880 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:36.011960 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc2cd1fda310466782c13f6d6e6728e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2cd1fda310466782c13f6d6e6728e5" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 43871 } } }
I20260928 15:07:36.012267 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5 [sys.catalog]: This master's current role is: LEADER
W20260928 15:07:36.012548 15893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:36.012606 15893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:36.012698 15896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:36.013005 15896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:36.014816 15896 catalog_manager.cc:1347] Generated new cluster ID: a31b59127fb34b1198af4dd4e020b162
I20260928 15:07:36.014881 15896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:36.017658 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.011781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51757 (local address 127.25.123.62:43871)
0928 15:07:36.012025 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.012036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.012335 (+ 299us) server_negotiation.cc:407] Connection header received
0928 15:07:36.012515 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.012522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.012582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.012760 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:36.013040 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.014422 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.015403 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.015839 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.015973 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.016751 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:36.016755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:36.016756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:36.016953 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:36.016986 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:36.016997 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:36.017075 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.017148 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.017313 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260928 15:07:36.018771 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:36.024981 15896 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:36.025763 15896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:36.028708 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:36.020592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55159 (local address 127.25.123.62:43871)
0928 15:07:36.020637 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:36.020640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:36.024677 (+ 4037us) server_negotiation.cc:407] Connection header received
0928 15:07:36.024720 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:36.024721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:36.024735 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:36.024877 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:36.025041 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.025797 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.026905 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:36.027267 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:36.027358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:36.028216 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:36.028218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:36.028218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:36.028224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:36.028240 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:36.028245 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:36.028307 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:36.028563 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 15:07:36.028620 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:07:36.034477 15896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc2cd1fda310466782c13f6d6e6728e5: Generated new TSK 0
I20260928 15:07:36.034716 15896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:36.037405 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15807
2026-09-28T15:07:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:36 dist-test-slave-qnz0 krb5kdc[15906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:36 dist-test-slave-qnz0 krb5kdc[15906](info): set up 2 sockets
Sep 28 15:07:36 dist-test-slave-qnz0 krb5kdc[15906](info): commencing operation
krb5kdc: starting...
W20260928 15:07:37.135686 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:37 dist-test-slave-qnz0 krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:37.259549 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45287
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:37.343760 15919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:37.343991 15919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:37.344017 15919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:37.345248 15919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:37.345302 15919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:37.345317 15919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:37.345336 15919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:37.345348 15919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:37.347093 15919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:37.347383 15919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:37.347669 15919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:37.352993 15925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:37.353096 15919 server_base.cc:1034] running on GCE node
W20260928 15:07:37.352996 15927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:37.352993 15924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:37.353508 15919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:37.353776 15919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:37.354964 15919 hybrid_clock.cc:648] HybridClock initialized: now 1790608057354942 us; error 46 us; skew 500 ppm
Sep 28 15:07:37 dist-test-slave-qnz0 krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608057, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:37.366226 15919 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:37.366885 15919 webserver.cc:460] Webserver started at http://127.25.123.62:46479/ using document root <none> and password file <none>
I20260928 15:07:37.367146 15919 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:37.367220 15919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:37.367381 15919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:37.368623 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "118bc5f0b2d24e499f120f17acf2009a"
format_stamp: "Formatted at 2026-09-28 15:07:37 on dist-test-slave-qnz0"
server_key: "bc85cc0b2e1f57bb7f00154fc4b0d47c"
server_key_iv: "7adfb969fb87e2933f25abfb7c03c071"
server_key_version: "encryptionkey@0"
I20260928 15:07:37.369016 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "118bc5f0b2d24e499f120f17acf2009a"
format_stamp: "Formatted at 2026-09-28 15:07:37 on dist-test-slave-qnz0"
server_key: "bc85cc0b2e1f57bb7f00154fc4b0d47c"
server_key_iv: "7adfb969fb87e2933f25abfb7c03c071"
server_key_version: "encryptionkey@0"
I20260928 15:07:37.370556 15919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:37.371434 15934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:37.371809 15919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:37.371943 15919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "118bc5f0b2d24e499f120f17acf2009a"
format_stamp: "Formatted at 2026-09-28 15:07:37 on dist-test-slave-qnz0"
server_key: "bc85cc0b2e1f57bb7f00154fc4b0d47c"
server_key_iv: "7adfb969fb87e2933f25abfb7c03c071"
server_key_version: "encryptionkey@0"
I20260928 15:07:37.372035 15919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:37.375607 15919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:37.377844 15919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:37.378005 15919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:37.382220 15919 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:45287
I20260928 15:07:37.382263 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:45287 every 8 connection(s)
I20260928 15:07:37.382588 15919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:37.383531 15987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:37.384955 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 15919
I20260928 15:07:37.385043 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:37.385281 26092 external_mini_cluster.cc:1428] Setting key 96afe62104357d91552a3f65ee9afe56
I20260928 15:07:37.386487 15987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a: Bootstrap starting.
I20260928 15:07:37.387238 15987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:37.387599 15987 log.cc:824] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:37.388267 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.385847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42323 (local address 127.25.123.62:45287)
0928 15:07:37.386090 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:37.386099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:37.386135 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:07:37.386705 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:37.386714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:37.386745 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:37.386920 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:37.387735 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:37.387739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:37.387740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:37.387745 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:37.387794 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:37.387804 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:37.387914 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:37.387977 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 15:07:37.388032 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260928 15:07:37.388443 15987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a: No bootstrap required, opened a new log
I20260928 15:07:37.389279 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:37.390863 15987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118bc5f0b2d24e499f120f17acf2009a" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45287 } }
I20260928 15:07:37.391074 15987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:07:37.391130 15987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118bc5f0b2d24e499f120f17acf2009a, State: Initialized, Role: FOLLOWER
I20260928 15:07:37.391279 15987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118bc5f0b2d24e499f120f17acf2009a" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45287 } }
I20260928 15:07:37.391374 15987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:07:37.391399 15987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:07:37.391714 15987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:37.392148 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:37.390962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48915 (local address 127.25.123.62:45287)
0928 15:07:37.391002 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:37.391004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:37.391070 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:07:37.391172 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:37.391173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:37.391179 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:37.391240 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:37.391849 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:37.391850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:37.391851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:07:37.391854 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:37.391874 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:07:37.391878 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:37.391942 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:37.392003 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 15:07:37.392018 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:07:37.392360 15987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118bc5f0b2d24e499f120f17acf2009a" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45287 } }
I20260928 15:07:37.392545 15987 leader_election.cc:302] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118bc5f0b2d24e499f120f17acf2009a; no voters:
I20260928 15:07:37.392712 15987 leader_election.cc:288] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:07:37.392746 15997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:37.392912 15997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [term 1 LEADER]: Becoming Leader. State: Replica: 118bc5f0b2d24e499f120f17acf2009a, State: Running, Role: LEADER
I20260928 15:07:37.393003 15987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:37.393052 15997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118bc5f0b2d24e499f120f17acf2009a" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45287 } }
I20260928 15:07:37.393424 15998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118bc5f0b2d24e499f120f17acf2009a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118bc5f0b2d24e499f120f17acf2009a" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45287 } } }
I20260928 15:07:37.393438 15999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118bc5f0b2d24e499f120f17acf2009a. Latest consensus state: current_term: 1 leader_uuid: "118bc5f0b2d24e499f120f17acf2009a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118bc5f0b2d24e499f120f17acf2009a" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 45287 } } }
I20260928 15:07:37.393617 15999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:37.393815 15998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:37.394037 16005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:37.394555 16005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:37.396030 16005 catalog_manager.cc:1347] Generated new cluster ID: cdca1eafd1964588b661a9f74d526cf2
I20260928 15:07:37.396091 16005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:37.407856 16005 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:37.408046 16005 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:37.422209 16005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 118bc5f0b2d24e499f120f17acf2009a: Generated new TSK 0
I20260928 15:07:37.422557 16005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:07:37.432749 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 15919
2026-09-28T15:07:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1394 ms)
[----------] 18 tests from AuthTokenIssuingTest (25027 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:37 dist-test-slave-qnz0 krb5kdc[16018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:37 dist-test-slave-qnz0 krb5kdc[16018](info): set up 2 sockets
Sep 28 15:07:37 dist-test-slave-qnz0 krb5kdc[16018](info): commencing operation
krb5kdc: starting...
W20260928 15:07:38.523582 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:38 dist-test-slave-qnz0 krb5kdc[16018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:38.652633 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:32865
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:32865,127.25.123.61:40995,127.25.123.60:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:38.740790 16031 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:07:38.741019 16031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:38.741037 16031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:38.741060 16031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:38.742280 16031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:38.742322 16031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:38.742337 16031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:38.742350 16031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:38.742384 16031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:38.743957 16031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:32865,127.25.123.61:40995,127.25.123.60:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:32865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:38.744231 16031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:38.744500 16031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:38.749789 16039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:38.749789 16037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:38.749823 16036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:38.750273 16031 server_base.cc:1034] running on GCE node
I20260928 15:07:38.750563 16031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:38.750881 16031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:38.752120 16031 hybrid_clock.cc:648] HybridClock initialized: now 1790608058752093 us; error 51 us; skew 500 ppm
Sep 28 15:07:38 dist-test-slave-qnz0 krb5kdc[16018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:38.754891 16031 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:38.755375 16031 webserver.cc:460] Webserver started at http://127.25.123.62:32807/ using document root <none> and password file <none>
I20260928 15:07:38.755633 16031 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:38.755700 16031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:38.755847 16031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:38.757041 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "d657bac607d944edbc44e4be04cb2386"
format_stamp: "Formatted at 2026-09-28 15:07:38 on dist-test-slave-qnz0"
server_key: "6610ab400ec023de53b1d14f2042fa47"
server_key_iv: "6e42f61c8577185a20840bc31a0fc4d1"
server_key_version: "encryptionkey@0"
I20260928 15:07:38.757411 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "d657bac607d944edbc44e4be04cb2386"
format_stamp: "Formatted at 2026-09-28 15:07:38 on dist-test-slave-qnz0"
server_key: "6610ab400ec023de53b1d14f2042fa47"
server_key_iv: "6e42f61c8577185a20840bc31a0fc4d1"
server_key_version: "encryptionkey@0"
I20260928 15:07:38.758846 16031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:38.759778 16046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:38.760005 16031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:38.760123 16031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "d657bac607d944edbc44e4be04cb2386"
format_stamp: "Formatted at 2026-09-28 15:07:38 on dist-test-slave-qnz0"
server_key: "6610ab400ec023de53b1d14f2042fa47"
server_key_iv: "6e42f61c8577185a20840bc31a0fc4d1"
server_key_version: "encryptionkey@0"
I20260928 15:07:38.760205 16031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:38.789485 16031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:38.791627 16031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:38.791807 16031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:38.795928 16031 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:32865
I20260928 15:07:38.795966 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:32865 every 8 connection(s)
I20260928 15:07:38.796348 16031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:38.797582 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:38.798249 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 16031
I20260928 15:07:38.798344 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:38.798557 26092 external_mini_cluster.cc:1428] Setting key 4c3a816a24ea09f4799bfb650a68d06d
I20260928 15:07:38.799610 16099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.123.61@KRBTEST.COM; defaulting to no policy
Sep 28 15:07:38 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.61@KRBTEST.COM" created.
I20260928 15:07:38.812198 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.800565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.25.123.62:32865)
0928 15:07:38.800733 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:38.800739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:38.800800 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:07:38.801343 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:38.801345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:38.801370 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:38.801466 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:38.802032 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.802850 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.803743 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.804055 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.804204 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:38.806637 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:38.806655 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:38.806655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:38.806664 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:38.808714 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:38.810753 (+ 2039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:38.810757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:38.810757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:38.810858 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:38.811220 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:38.811221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:38.811221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:38.811532 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:38.811608 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:38.811903 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 15:07:38.812018 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260928 15:07:38.812353 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.800452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:32865 (local address 127.0.0.1:46326)
0928 15:07:38.800664 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.800709 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:38.801174 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:38.801471 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:38.801473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:38.801766 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:38.801923 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:38.801928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.802954 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.802955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:38.803641 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:38.803643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.804174 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.804176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:38.804239 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:38.804879 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:38.804887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:38.806529 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:38.808811 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:38.808822 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:38.808824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:38.809172 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:38.810972 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:38.810973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:38.810974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:38.811128 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:38.811620 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:38.811628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:38.811720 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 15:07:38.811862 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 15:07:38.812145 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":74,"thread_start_us":92,"threads_started":1}
I20260928 15:07:38.813380 16099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:38.813858 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.813691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:44966)
0928 15:07:38.813783 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.813834 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 15:07:38.814055 16048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
I20260928 15:07:38.814641 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.814526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:44982)
0928 15:07:38.814588 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.814619 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:07:38.814756 16099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:40995: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab.
Entry for principal kudu/127.25.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab.
I20260928 15:07:38.829857 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:40995
--webserver_interface=127.25.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab
--principal=kudu/127.25.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:32865,127.25.123.61:40995,127.25.123.60:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:07:38.864013 16099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } attempt: 1
I20260928 15:07:38.864594 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.864455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:44992)
0928 15:07:38.864517 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.864561 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 15:07:38.864997 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.864891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:44998)
0928 15:07:38.864942 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.864976 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 15:07:38.865087 16099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:40995: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
W20260928 15:07:38.920063 16106 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:07:38.920318 16106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:38.920337 16106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:38.920360 16106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:38.922394 16106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:38.922457 16106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:38.922472 16106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:38.922485 16106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:38.922498 16106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:38.924742 16106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:32865,127.25.123.61:40995,127.25.123.60:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab
--principal=kudu/127.25.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:40995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:38.925067 16106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:38.925312 16106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:38.932735 16111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:38.932822 16106 server_base.cc:1034] running on GCE node
W20260928 15:07:38.933034 16114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:38.932730 16112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:38.933208 16099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } attempt: 2
I20260928 15:07:38.933364 16106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:38.933640 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.933506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:45006)
0928 15:07:38.933576 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.933606 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:07:38.933741 16106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:38.934032 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.933918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:45018)
0928 15:07:38.933986 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.934010 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:07:38.934157 16099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:40995: Network error: Client connection negotiation failed: client connection to 127.25.123.61:40995: connect: Connection refused (error 111)
I20260928 15:07:38.934947 16106 hybrid_clock.cc:648] HybridClock initialized: now 1790608058934931 us; error 27 us; skew 500 ppm
Sep 28 15:07:38 dist-test-slave-qnz0 krb5kdc[16018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:38.937955 16106 init.cc:377] Logged in from keytab as kudu/127.25.123.61@KRBTEST.COM (short username kudu)
I20260928 15:07:38.938529 16106 webserver.cc:460] Webserver started at http://127.25.123.61:36637/ using document root <none> and password file <none>
I20260928 15:07:38.938797 16106 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:38.938874 16106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:38.939029 16106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:38.940141 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data/instance:
uuid: "e785185e5a33449bae11b5963fbcf50d"
format_stamp: "Formatted at 2026-09-28 15:07:38 on dist-test-slave-qnz0"
server_key: "f9fb2bd78fad38c5c8461c4465dec1f2"
server_key_iv: "65f1e657f5af0a9e860fcc0ecda3d057"
server_key_version: "encryptionkey@0"
I20260928 15:07:38.940552 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/wal/instance:
uuid: "e785185e5a33449bae11b5963fbcf50d"
format_stamp: "Formatted at 2026-09-28 15:07:38 on dist-test-slave-qnz0"
server_key: "f9fb2bd78fad38c5c8461c4465dec1f2"
server_key_iv: "65f1e657f5af0a9e860fcc0ecda3d057"
server_key_version: "encryptionkey@0"
I20260928 15:07:38.942047 16106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:38.942924 16121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:38.943194 16106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 15:07:38.943315 16106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/wal
uuid: "e785185e5a33449bae11b5963fbcf50d"
format_stamp: "Formatted at 2026-09-28 15:07:38 on dist-test-slave-qnz0"
server_key: "f9fb2bd78fad38c5c8461c4465dec1f2"
server_key_iv: "65f1e657f5af0a9e860fcc0ecda3d057"
server_key_version: "encryptionkey@0"
I20260928 15:07:38.943432 16106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:38.955137 16106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:38.957818 16106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:38.958168 16106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:38.962843 16106 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.61:40995
I20260928 15:07:38.962911 16173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.61:40995 every 8 connection(s)
I20260928 15:07:38.963372 16106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
I20260928 15:07:38.964671 16174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:38.965893 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 16106
I20260928 15:07:38.966022 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-1/wal/instance
I20260928 15:07:38.966203 26092 external_mini_cluster.cc:1428] Setting key d3d101fda58712efe26c366e4ff4ebd8
I20260928 15:07:38.966863 16174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.123.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.60@KRBTEST.COM" created.
Sep 28 15:07:38 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:38.983903 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.967931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.25.123.62:32865)
0928 15:07:38.967994 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:38.967996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:38.968200 (+ 204us) server_negotiation.cc:407] Connection header received
0928 15:07:38.970108 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:38.970111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:38.970155 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:38.970305 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:38.971493 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.972498 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.974308 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.974714 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.974821 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:38.980787 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:38.980795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:38.980796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:38.980805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:38.982333 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:38.982839 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:38.982841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:38.982841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:38.982908 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:38.983185 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:38.983185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:38.983186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:38.983326 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:38.983401 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:38.983704 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 15:07:38.983802 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 15:07:38.984127 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.967807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:32865 (local address 127.0.0.1:46332)
0928 15:07:38.968048 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.968093 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:38.969420 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:38.970444 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:38.970450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:38.971065 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:38.971317 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:38.971324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.972832 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.972834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:38.974054 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:38.974061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.974899 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.974900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:38.975057 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:38.976147 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:38.976160 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:38.980650 (+ 4490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:38.982479 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:38.982490 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:38.982492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:38.982760 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:38.982988 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:38.982988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:38.982989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:38.983111 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:38.983404 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:38.983412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:38.983570 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 15:07:38.983683 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 15:07:38.983861 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260928 15:07:38.984966 16174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:07:38 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:38.994510 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.985351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:45030)
0928 15:07:38.985386 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.985397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:38.985436 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:38.985688 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:38.985689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:38.985852 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:38.985901 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:38.985902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.986788 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.986789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:38.987284 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:38.987286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.987826 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.987827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:38.987894 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:38.988702 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:38.988709 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:38.990508 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:38.992760 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:38.992761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:38.992762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:38.992928 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:38.993175 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:38.993175 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:38.993176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:38.993231 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:38.993854 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:38.993856 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:38.993918 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 15:07:38.994002 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:07:38.994410 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":277}
I20260928 15:07:38.994524 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.985364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.25.123.61:40995)
0928 15:07:38.985544 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:38.985549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:38.985576 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:07:38.985594 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:38.985596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:38.985607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:38.985688 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:38.985975 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.986691 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.987371 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:38.987698 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:38.987827 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:38.990658 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:38.990678 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:38.990678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:38.990686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:38.992650 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:38.993044 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:38.993048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:38.993049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:38.993105 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:38.993304 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:38.993305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:38.993305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:38.993609 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:38.993959 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:38.994044 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 15:07:38.994367 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260928 15:07:38.995626 16174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:38.996086 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.995955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52790)
0928 15:07:38.996010 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.996061 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 15:07:38.996279 16124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
I20260928 15:07:38.996871 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:38.996734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52804)
0928 15:07:38.996802 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:38.996836 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:07:38.997031 16174 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:45285: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab.
Entry for principal kudu/127.25.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab.
I20260928 15:07:39.000924 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:45285
--webserver_interface=127.25.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab
--principal=kudu/127.25.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:32865,127.25.123.61:40995,127.25.123.60:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:07:39.025306 16099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } attempt: 3
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:39.033490 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.025773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:45034)
0928 15:07:39.025822 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.025838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:39.025906 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:39.026076 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:39.026077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:39.026263 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:39.026353 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.026355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.027092 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.027093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.027921 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.027923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.028471 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.028473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.028559 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.029081 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:39.029087 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:39.030547 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:39.032298 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.032299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.032299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.032552 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.032769 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.032770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.032770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.032839 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.033128 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:39.033130 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:39.033183 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 15:07:39.033269 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:07:39.033379 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 15:07:39.033484 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.025875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.25.123.61:40995)
0928 15:07:39.025942 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.025949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.025960 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:39.025988 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.025989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.025999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.026055 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.026435 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.027011 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.028028 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.028286 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.028353 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.030655 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.030657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.030657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.030672 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.032185 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.032627 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.032628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.032629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.032691 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.032909 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.032910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.032910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.033050 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.033128 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.033294 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.033379 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 15:07:39.034289 16099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:39.034675 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.034558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52814)
0928 15:07:39.034617 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.034651 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 15:07:39.035065 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.034991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52816)
0928 15:07:39.035027 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.035050 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 15:07:39.035171 16099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:45285: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
I20260928 15:07:39.046296 16174 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } attempt: 1
I20260928 15:07:39.046875 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.046671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52820)
0928 15:07:39.046751 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.046819 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:07:39.047269 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.047157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52836)
0928 15:07:39.047210 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.047240 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 15:07:39.047410 16174 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:45285: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
I20260928 15:07:39.066352 16099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } attempt: 1
I20260928 15:07:39.066977 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.066759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52846)
0928 15:07:39.066872 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.066918 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:07:39.067714 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.067550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52848)
0928 15:07:39.067636 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.067681 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:07:39.067905 16099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:45285: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
W20260928 15:07:39.088356 16181 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:07:39.088689 16181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:39.088726 16181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:39.088770 16181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:39.090054 16181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:39.090123 16181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:39.090147 16181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:39.090168 16181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:39.090198 16181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:39.091729 16181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:32865,127.25.123.61:40995,127.25.123.60:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab
--principal=kudu/127.25.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:45285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:39.092034 16181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:39.092281 16181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:39.097301 16189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:39.097301 16186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:39.097494 16187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:39.097800 16181 server_base.cc:1034] running on GCE node
I20260928 15:07:39.098066 16181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:39.098387 16181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:39.099556 16181 hybrid_clock.cc:648] HybridClock initialized: now 1790608059099501 us; error 69 us; skew 500 ppm
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608059, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:39.102291 16181 init.cc:377] Logged in from keytab as kudu/127.25.123.60@KRBTEST.COM (short username kudu)
I20260928 15:07:39.102872 16181 webserver.cc:460] Webserver started at http://127.25.123.60:44401/ using document root <none> and password file <none>
I20260928 15:07:39.103142 16181 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:39.103221 16181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:39.103386 16181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:39.104725 16181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data/instance:
uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754"
format_stamp: "Formatted at 2026-09-28 15:07:39 on dist-test-slave-qnz0"
server_key: "c25eb7f031591971669e11562b3edc8b"
server_key_iv: "f94f9e4cd7e62cb6c1abc5514f66b155"
server_key_version: "encryptionkey@0"
I20260928 15:07:39.105175 16181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/wal/instance:
uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754"
format_stamp: "Formatted at 2026-09-28 15:07:39 on dist-test-slave-qnz0"
server_key: "c25eb7f031591971669e11562b3edc8b"
server_key_iv: "f94f9e4cd7e62cb6c1abc5514f66b155"
server_key_version: "encryptionkey@0"
I20260928 15:07:39.106716 16181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:07:39.107825 16196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:39.108114 16181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:39.108248 16181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/wal
uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754"
format_stamp: "Formatted at 2026-09-28 15:07:39 on dist-test-slave-qnz0"
server_key: "c25eb7f031591971669e11562b3edc8b"
server_key_iv: "f94f9e4cd7e62cb6c1abc5514f66b155"
server_key_version: "encryptionkey@0"
I20260928 15:07:39.108430 16181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:39.115585 16174 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } attempt: 2
I20260928 15:07:39.116086 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.115951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52854)
0928 15:07:39.116016 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.116057 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 15:07:39.116503 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.116373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52860)
0928 15:07:39.116443 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.116473 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:07:39.116602 16174 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:45285: Network error: Client connection negotiation failed: client connection to 127.25.123.60:45285: connect: Connection refused (error 111)
I20260928 15:07:39.121246 16181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:39.123492 16181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:39.123663 16181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:39.129029 16181 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.60:45285
I20260928 15:07:39.129087 16248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.60:45285 every 8 connection(s)
I20260928 15:07:39.129467 16181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
I20260928 15:07:39.130632 16249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:39.132225 16249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:39.136685 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 16181
I20260928 15:07:39.136825 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790607748072058-26092-0/minicluster-data/master-2/wal/instance
I20260928 15:07:39.137034 26092 external_mini_cluster.cc:1428] Setting key e8749dda1b73335b4cb43b7c0114f6a1
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608059, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:07:39.143085 16099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } attempt: 2
I20260928 15:07:39.143285 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.133214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.25.123.62:32865)
0928 15:07:39.133463 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.133470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.133695 (+ 225us) server_negotiation.cc:407] Connection header received
0928 15:07:39.134290 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.134291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.134306 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.134362 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.135232 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.135741 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.136927 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.137161 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.137230 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.140045 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.140047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.140048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.140057 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.141683 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.142131 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.142132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.142133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.142196 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.142460 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.142461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.142461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.142600 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.142666 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.142952 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.143041 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260928 15:07:39.143307 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.133162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:32865 (local address 127.0.0.1:46334)
0928 15:07:39.133540 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.133594 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:39.134160 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:39.134438 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:39.134442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:39.134862 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:39.135087 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.135092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.135844 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.135846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.136808 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.136811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.137283 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.137284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.137455 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.138109 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:39.138126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:39.139899 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:39.141782 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.141788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.141791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.142045 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.142266 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.142267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.142267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.142383 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.142670 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:39.142676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:39.142817 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 15:07:39.142908 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:07:39.143039 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":145,"threads_started":1}
I20260928 15:07:39.144508 16249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:39.147078 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.137566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.25.123.60:45285)
0928 15:07:39.137780 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.137785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.137805 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:07:39.137826 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.137828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.137863 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.137956 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.138576 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.139711 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.140550 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.141115 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.141195 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.143708 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.143716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.143716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.143724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.145657 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.146092 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.146096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.146096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.146161 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.146361 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.146362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.146363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.146586 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.146748 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.146839 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.146918 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":120,"threads_started":1}
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608059, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:39.152707 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.143374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52876)
0928 15:07:39.143579 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.143590 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:39.143636 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:39.143973 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:39.143974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:39.144467 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:39.144518 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.144520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.145328 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.145329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.146143 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.146146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.146545 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.146546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.146605 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.147031 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:39.147037 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:39.148846 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:39.151580 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.151581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.151582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.151772 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.152016 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.152016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.152017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.152078 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.152359 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:39.152360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:39.152430 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:07:39.152493 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 15:07:39.152591 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260928 15:07:39.152731 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.143665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.25.123.60:45285)
0928 15:07:39.143844 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.143846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.143852 (+ 6us) server_negotiation.cc:407] Connection header received
0928 15:07:39.143869 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.143869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.143878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.143954 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.144605 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.145242 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.146204 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.146465 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.146548 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.148966 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.148968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.148968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.148975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.151441 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.151846 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.151848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.151848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.151932 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.152153 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.152154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.152154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.152282 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.152367 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.152527 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.152601 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":118,"threads_started":1}
I20260928 15:07:39.153115 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.144947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.25.123.61:40995)
0928 15:07:39.145001 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.145004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.145013 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:39.145038 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.145039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.145049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.145098 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.145387 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.146090 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.147003 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.147295 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.147358 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.149722 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.149723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.149724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.149732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.151865 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.152237 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.152238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.152239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.152304 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.152560 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.152561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.152561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.152705 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.152761 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.152939 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.153032 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 15:07:39.153123 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.144789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:40995 (local address 127.0.0.1:45050)
0928 15:07:39.144833 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.144843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:39.144878 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:39.145145 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:39.145146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:39.145285 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:39.145323 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.145324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.146162 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.146163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.146917 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.146919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.147408 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.147409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.147460 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.147800 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:39.147806 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:39.149621 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:39.151953 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.151954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.151955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.152166 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.152427 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.152428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.152428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.152492 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.152786 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:39.152787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:39.152829 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:07:39.152895 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 15:07:39.153006 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 15:07:39.153770 16249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:39.155316 16099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386: Bootstrap starting.
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:39.156080 16099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:39.156469 16099 log.cc:824] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:39.157403 16099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386: No bootstrap required, opened a new log
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608059, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:07:39.159019 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.149049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.25.123.62:32865)
0928 15:07:39.149095 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.149097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.149108 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:39.149124 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.149125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.149161 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.149230 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.149767 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.151375 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.152692 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.153005 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.153049 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.156356 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.156358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.156358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.156366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.157904 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.158262 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.158263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.158264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.158320 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.158514 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.158515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.158515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.158624 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.158788 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.158831 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.158915 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 15:07:39.160004 16099 raft_consensus.cc:348] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.160192 16099 raft_consensus.cc:729] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d657bac607d944edbc44e4be04cb2386, State: Initialized, Role: FOLLOWER
I20260928 15:07:39.160296 16099 consensus_queue.cc:260] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.160737 16099 sys_catalog.cc:564] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:39.161250 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.154045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52888)
0928 15:07:39.154086 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.154096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:39.154133 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:39.154250 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:39.154251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:39.154392 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:39.154420 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.154422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.155120 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.155121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.155821 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.155824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.156236 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.156237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.156288 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.156764 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:39.156769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:39.158222 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:39.160101 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.160102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.160103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.160308 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.160558 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.160559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.160559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.160610 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.160939 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:39.160940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:39.160982 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:07:39.161025 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 15:07:39.161120 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 15:07:39.161681 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.154068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.25.123.60:45285)
0928 15:07:39.154111 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.154113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.154121 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:07:39.154188 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.154189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.154197 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.154248 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.154501 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.155031 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.155895 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.156113 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.156487 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.158312 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.158313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.158313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.158319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.160007 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.160425 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.160426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.160427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.160485 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.160674 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.160675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.160675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.160853 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.161465 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.161508 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.161599 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 15:07:39.162201 16256 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.162295 16256 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:39.163113 16249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: Bootstrap starting.
I20260928 15:07:39.163777 16249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:39.164096 16249 log.cc:824] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:39.164935 16249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: No bootstrap required, opened a new log
I20260928 15:07:39.167256 16249 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.167413 16249 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fbf5aa0abe4ee3b947e1ef4ee49754, State: Initialized, Role: FOLLOWER
I20260928 15:07:39.167503 16249 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.167830 16249 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:07:39.168272 16267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:39.168381 16267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:39.168902 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.159715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.25.123.61:40995)
0928 15:07:39.159755 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.159757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.159804 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:07:39.159877 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.159877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.159887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.159935 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.160502 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.161048 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.162112 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.162418 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.162492 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.165116 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.165118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.165118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.165126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.166936 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.167539 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.167541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.167541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.167623 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.167835 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.167836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.167836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.167981 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.168134 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.168724 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.168807 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 15:07:39.167034 16031 master_runner.cc:186] Error getting master registration for 127.25.123.61:40995: OK, instance_id { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" instance_seqno: 1790608058961636 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:07:39.170112 16031 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:07:39.167817 16269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.170308 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:39.170323 16269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:07:39.169952 16281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:39.170724 16281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:39.171308 16073 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d657bac607d944edbc44e4be04cb2386"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348
I20260928 15:07:39.171429 16073 raft_consensus.cc:482] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:07:39.171475 16073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:39.172315 16073 raft_consensus.cc:504] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.172602 16073 leader_election.cc:288] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [CANDIDATE]: Term 1 election: Requested vote from peers e785185e5a33449bae11b5963fbcf50d (127.25.123.61:40995), c1fbf5aa0abe4ee3b947e1ef4ee49754 (127.25.123.60:45285)
I20260928 15:07:39.172910 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d657bac607d944edbc44e4be04cb2386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754"
I20260928 15:07:39.173048 16223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:39.173969 16223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d657bac607d944edbc44e4be04cb2386 in term 1.
I20260928 15:07:39.171918 16181 master_runner.cc:186] Error getting master registration for 127.25.123.61:40995: OK, instance_id { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" instance_seqno: 1790608058961636 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:07:39.174213 16181 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:07:39.174309 16049 leader_election.cc:302] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1fbf5aa0abe4ee3b947e1ef4ee49754, d657bac607d944edbc44e4be04cb2386; no voters:
I20260928 15:07:39.175884 16256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:39.176239 16256 raft_consensus.cc:686] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [term 1 LEADER]: Becoming Leader. State: Replica: d657bac607d944edbc44e4be04cb2386, State: Running, Role: LEADER
I20260928 15:07:39.176512 16256 consensus_queue.cc:237] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.177503 16256 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657bac607d944edbc44e4be04cb2386. Latest consensus state: current_term: 1 leader_uuid: "d657bac607d944edbc44e4be04cb2386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.177731 16256 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:39.183941 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.175717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.25.123.62:32865)
0928 15:07:39.175756 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.175758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.175927 (+ 169us) server_negotiation.cc:407] Connection header received
0928 15:07:39.176030 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.176031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.176042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.176072 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.176913 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.177628 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.178726 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.179029 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.179110 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.180857 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.180858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.180859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.180866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.182715 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.183110 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.183111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.183112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.183185 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.183385 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.183386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.183387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.183522 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.183638 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.183759 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.183843 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:07:39.192720 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.175597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.25.123.60:45285)
0928 15:07:39.175636 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.175639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.184053 (+ 8414us) server_negotiation.cc:407] Connection header received
0928 15:07:39.184153 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.184154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.184166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.184198 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.184931 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.185670 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.187163 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.187490 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.187533 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.189430 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.189432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.189432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.189439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.191304 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.191690 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.191691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.191692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.191745 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.191937 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.191938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.191938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.192085 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.192188 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.192419 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.192601 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:07:39.193225 16291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:39.193450 16291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:39.194231 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.177459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.25.123.61:40995)
0928 15:07:39.177502 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.177504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.184596 (+ 7092us) server_negotiation.cc:407] Connection header received
0928 15:07:39.184705 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.184706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.184719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.184752 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.185356 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.186088 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.187323 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.187778 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.187831 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.190226 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.190228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.190228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.190236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.191850 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.192414 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.192415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.192415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.192474 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.192776 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.192777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.192777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.192925 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.193012 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.194050 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.194132 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:07:39.195434 16223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [term 1 FOLLOWER]: Refusing update from remote peer d657bac607d944edbc44e4be04cb2386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:39.195809 16256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:39.197261 16269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657bac607d944edbc44e4be04cb2386. Latest consensus state: current_term: 1 leader_uuid: "d657bac607d944edbc44e4be04cb2386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.197374 16269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:39.205386 16296 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d657bac607d944edbc44e4be04cb2386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.206223 16269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d657bac607d944edbc44e4be04cb2386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.206386 16269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:39.205811 16292 mvcc.cc:205] Tried to move back new op lower bound from 7334330610460958720 to 7334330610388918272. Current Snapshot: MvccSnapshot[applied={T|T < 7334330610460958720}]
I20260928 15:07:39.206545 16296 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:39.207336 16291 catalog_manager.cc:1347] Generated new cluster ID: 6eb84ca3dd7d413596a4e24f5a96706f
I20260928 15:07:39.207492 16291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:39.207727 16174 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } attempt: 3
Sep 28 15:07:39 dist-test-slave-qnz0 krb5kdc[16018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608058, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:07:39.215788 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.208021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:45285 (local address 127.0.0.1:52896)
0928 15:07:39.208120 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:39.208133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:39.208173 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:39.208295 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:39.208296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:39.208521 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:39.208569 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.208571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.209503 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.209504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.210297 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:39.210298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.210804 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.210805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:39.210869 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.211530 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:39.211535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:39.213097 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:39.214836 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.214837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.214838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.215033 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.215197 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:39.215197 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:39.215198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:39.215242 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:39.215502 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:39.215503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:39.215544 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:07:39.215597 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 15:07:39.215689 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:07:39.216724 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:39.208121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.25.123.60:45285)
0928 15:07:39.208173 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.208175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.208185 (+ 10us) server_negotiation.cc:407] Connection header received
0928 15:07:39.208215 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.208216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.208227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.208290 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.208657 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.209413 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.210396 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:39.210692 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:39.210742 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:39.213188 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:39.213189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:39.213190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:39.213197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:39.214745 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.215097 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.215098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.215098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.215144 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:39.215308 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:39.215308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:39.215309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:39.215429 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:39.215481 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:39.216197 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 15:07:39.216634 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 15:07:39.218580 16174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: Bootstrap starting.
I20260928 15:07:39.219358 16174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:39.219663 16174 log.cc:824] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:39.220718 16174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: No bootstrap required, opened a new log
I20260928 15:07:39.224447 16174 raft_consensus.cc:348] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.224855 16174 raft_consensus.cc:729] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e785185e5a33449bae11b5963fbcf50d, State: Initialized, Role: FOLLOWER
I20260928 15:07:39.225080 16174 consensus_queue.cc:260] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } }
I20260928 15:07:39.225625 16300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.225725 16300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:39.226006 16147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:39.227025 16147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [term 1 FOLLOWER]: Refusing update from remote peer d657bac607d944edbc44e4be04cb2386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:39.227401 16148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d657bac607d944edbc44e4be04cb2386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e785185e5a33449bae11b5963fbcf50d"
I20260928 15:07:39.227456 16297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:39.228139 16174 sys_catalog.cc:564] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:39.228204 16148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d657bac607d944edbc44e4be04cb2386 in term 1.
I20260928 15:07:39.229576 16300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657bac607d944edbc44e4be04cb2386. Latest consensus state: current_term: 1 leader_uuid: "d657bac607d944edbc44e4be04cb2386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.229676 16300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:39.229710 16300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d657bac607d944edbc44e4be04cb2386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.229761 16300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:39.230798 16306 mvcc.cc:205] Tried to move back new op lower bound from 7334330610460958720 to 7334330610388918272. Current Snapshot: MvccSnapshot[applied={T|T < 7334330610460958720}]
W20260928 15:07:39.232437 16314 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:39.232501 16314 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:39.236783 16297 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d657bac607d944edbc44e4be04cb2386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657bac607d944edbc44e4be04cb2386" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 32865 } } peers { permanent_uuid: "e785185e5a33449bae11b5963fbcf50d" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 40995 } } peers { permanent_uuid: "c1fbf5aa0abe4ee3b947e1ef4ee49754" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 45285 } } }
I20260928 15:07:39.236902 16297 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:39.239476 16291 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:39.240262 16291 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:39.248852 16291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d657bac607d944edbc44e4be04cb2386: Generated new TSK 0
I20260928 15:07:39.249089 16291 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:07:39.284731 16103 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.283589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.25.123.62:32865)
0928 15:07:39.283640 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.283642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.283951 (+ 309us) server_negotiation.cc:407] Connection header received
0928 15:07:39.284065 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.284067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.284079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.284112 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.284652 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46366: BlockingRecv error: recv got EOF from 127.0.0.1:46366 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 15:07:39.290017 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.288741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.25.123.61:40995)
0928 15:07:39.288827 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.288829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.289165 (+ 336us) server_negotiation.cc:407] Connection header received
0928 15:07:39.289404 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.289406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.289422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.289514 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.289943 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45082: BlockingRecv error: recv got EOF from 127.0.0.1:45082 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 15:07:39.290956 16254 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.288998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.25.123.60:45285)
0928 15:07:39.289037 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.289039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.290397 (+ 1358us) server_negotiation.cc:407] Connection header received
0928 15:07:39.290501 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.290502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.290513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.290545 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.290892 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52904: BlockingRecv error: recv got EOF from 127.0.0.1:52904 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:07:39.293984 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.293141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.25.123.61:40995)
0928 15:07:39.293189 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.293191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.293396 (+ 205us) server_negotiation.cc:407] Connection header received
0928 15:07:39.293519 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.293520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.293540 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.293574 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.293929 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45084: BlockingRecv error: recv got EOF from 127.0.0.1:45084 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 15:07:39.297477 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.296505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.25.123.61:40995)
0928 15:07:39.296543 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.296545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.296851 (+ 306us) server_negotiation.cc:407] Connection header received
0928 15:07:39.296962 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.296963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.296976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.297008 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.297420 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45100: BlockingRecv error: recv got EOF from 127.0.0.1:45100 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:07:39.302086 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.301144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.25.123.61:40995)
0928 15:07:39.301188 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.301190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.301492 (+ 302us) server_negotiation.cc:407] Connection header received
0928 15:07:39.301602 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.301603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.301616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.301646 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.302026 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45114: BlockingRecv error: recv got EOF from 127.0.0.1:45114 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 15:07:39.308488 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.307483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.25.123.61:40995)
0928 15:07:39.307526 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.307528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.307839 (+ 311us) server_negotiation.cc:407] Connection header received
0928 15:07:39.307948 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.307949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.307961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.307991 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.308431 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45118: BlockingRecv error: recv got EOF from 127.0.0.1:45118 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 15:07:39.318907 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.317916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.25.123.61:40995)
0928 15:07:39.317974 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.317976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.318254 (+ 278us) server_negotiation.cc:407] Connection header received
0928 15:07:39.318363 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.318364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.318377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.318407 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.318831 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45128: BlockingRecv error: recv got EOF from 127.0.0.1:45128 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:07:39.337387 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.336411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.25.123.61:40995)
0928 15:07:39.336459 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.336462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.336759 (+ 297us) server_negotiation.cc:407] Connection header received
0928 15:07:39.336873 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.336874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.336886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.336916 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.337323 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45136: BlockingRecv error: recv got EOF from 127.0.0.1:45136 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 15:07:39.371688 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.370948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.25.123.61:40995)
0928 15:07:39.371055 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.371058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.371071 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:07:39.371124 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.371125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.371138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.371208 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.371622 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45148: BlockingRecv error: recv got EOF from 127.0.0.1:45148 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 15:07:39.437758 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.437073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.25.123.61:40995)
0928 15:07:39.437155 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.437157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.437172 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:07:39.437260 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.437261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.437271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.437323 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.437671 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45162: BlockingRecv error: recv got EOF from 127.0.0.1:45162 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 15:07:39.567956 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.567181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.25.123.61:40995)
0928 15:07:39.567289 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.567291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.567373 (+ 82us) server_negotiation.cc:407] Connection header received
0928 15:07:39.567438 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.567439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.567452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.567520 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.567894 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45170: BlockingRecv error: recv got EOF from 127.0.0.1:45170 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 15:07:39.826265 16179 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:39.825580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.25.123.61:40995)
0928 15:07:39.825696 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:39.825698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:39.825709 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:39.825738 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:39.825739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:39.825749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:39.825817 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:39.826215 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45176: BlockingRecv error: recv got EOF from 127.0.0.1:45176 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:07:40.171276 16281 catalog_manager.cc:1259] Loaded cluster ID: 6eb84ca3dd7d413596a4e24f5a96706f
I20260928 15:07:40.171428 16281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: loading cluster ID for follower catalog manager: success
I20260928 15:07:40.172271 16281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: acquiring CA information for follower catalog manager: success
I20260928 15:07:40.172616 16281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1fbf5aa0abe4ee3b947e1ef4ee49754: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:07:40.233011 16314 catalog_manager.cc:1259] Loaded cluster ID: 6eb84ca3dd7d413596a4e24f5a96706f
I20260928 15:07:40.233094 16314 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: loading cluster ID for follower catalog manager: success
I20260928 15:07:40.233970 16314 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: acquiring CA information for follower catalog manager: success
I20260928 15:07:40.234309 16314 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e785185e5a33449bae11b5963fbcf50d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:07:40.342738 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:40.339769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.25.123.61:40995)
0928 15:07:40.339869 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:40.339871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:40.339882 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:40.339935 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:40.339935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:40.339984 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:40.340058 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:40.340272 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.340959 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.341832 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.342030 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.342095 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:40.342284 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:40.342403 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:40.342548 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:40.342571 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 15:07:40.342593 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
I20260928 15:07:40.347899 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:40.344984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.25.123.60:45285)
0928 15:07:40.345109 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:40.345112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:40.345121 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:40.345167 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:40.345168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:40.345203 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:40.345276 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:40.345636 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.346139 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.347030 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.347229 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.347289 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:40.347468 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:40.347581 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:40.347683 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:40.347758 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 15:07:40.347781 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260928 15:07:40.352931 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:40.349831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.25.123.60:45285)
0928 15:07:40.349879 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:40.349880 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:40.350096 (+ 216us) server_negotiation.cc:407] Connection header received
0928 15:07:40.350122 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:40.350122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:40.350136 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:40.350194 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:40.350349 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.351072 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.352064 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.352331 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.352563 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:40.352591 (+ 28us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:40.352623 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:40.352667 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:40.352833 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:07:40.352856 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:07:40.352962 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:40.349741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.25.123.61:40995)
0928 15:07:40.349782 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:40.349784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:40.349940 (+ 156us) server_negotiation.cc:407] Connection header received
0928 15:07:40.350045 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:40.350046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:40.350057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:40.350084 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:40.350282 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.350944 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.352002 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.352305 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.352433 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:40.352590 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:40.352634 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:40.352702 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:40.352838 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 15:07:40.352870 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:07:40.353449 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:40.349945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.25.123.62:32865)
0928 15:07:40.350364 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:40.350366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:40.350374 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:07:40.350420 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:40.350421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:40.350473 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:40.350590 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:40.350732 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.351381 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.352227 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:40.352558 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:40.352615 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:40.352844 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:40.352961 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:40.353041 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:40.353333 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 15:07:40.353360 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":66,"threads_started":1}
I20260928 15:07:40.355316 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 16031
I20260928 15:07:40.361081 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 16106
I20260928 15:07:40.366261 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 16181
2026-09-28T15:07:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2934 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:07:40 dist-test-slave-qnz0 krb5kdc[16396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:07:40 dist-test-slave-qnz0 krb5kdc[16396](info): set up 2 sockets
Sep 28 15:07:40 dist-test-slave-qnz0 krb5kdc[16396](info): commencing operation
krb5kdc: starting...
W20260928 15:07:41.447644 26092 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/test-user.keytab.
Sep 28 15:07:41 dist-test-slave-qnz0 krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.123.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
Entry for principal kudu/127.25.123.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab.
I20260928 15:07:41.579895 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:40505
--webserver_interface=127.25.123.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:40505,127.25.123.61:41653,127.25.123.60:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:07:41.662197 16409 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:07:41.662482 16409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:41.662513 16409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:41.662556 16409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:41.663774 16409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:41.663862 16409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:41.663887 16409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:41.663909 16409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:41.663940 16409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:41.665544 16409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:40505,127.25.123.61:41653,127.25.123.60:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.62.keytab
--principal=kudu/127.25.123.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.62:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.123.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:41.665930 16409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:41.666224 16409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:41.671319 16415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:41.671319 16417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:41.671365 16414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:41.671628 16409 server_base.cc:1034] running on GCE node
I20260928 15:07:41.671885 16409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:41.672214 16409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:41.673398 16409 hybrid_clock.cc:648] HybridClock initialized: now 1790608061673378 us; error 56 us; skew 500 ppm
Sep 28 15:07:41 dist-test-slave-qnz0 krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:41.676683 16409 init.cc:377] Logged in from keytab as kudu/127.25.123.62@KRBTEST.COM (short username kudu)
I20260928 15:07:41.677224 16409 webserver.cc:460] Webserver started at http://127.25.123.62:43291/ using document root <none> and password file <none>
I20260928 15:07:41.677531 16409 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:41.677601 16409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:41.677762 16409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:41.678886 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data/instance:
uuid: "c480d253c1e445a5919d44aa4e9c6d73"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-qnz0"
server_key: "91f694da38a7c89bfb2c6f943f02bf56"
server_key_iv: "007634c4fb61c66ec6a5a3c6d65b9827"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.679255 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/wal/instance:
uuid: "c480d253c1e445a5919d44aa4e9c6d73"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-qnz0"
server_key: "91f694da38a7c89bfb2c6f943f02bf56"
server_key_iv: "007634c4fb61c66ec6a5a3c6d65b9827"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.680842 16409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 15:07:41.681751 16424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:41.682045 16409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 15:07:41.682189 16409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/wal
uuid: "c480d253c1e445a5919d44aa4e9c6d73"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-qnz0"
server_key: "91f694da38a7c89bfb2c6f943f02bf56"
server_key_iv: "007634c4fb61c66ec6a5a3c6d65b9827"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.682394 16409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:41.697953 16409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:41.700529 16409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:41.700714 16409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:41.704845 16409 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.62:40505
I20260928 15:07:41.704900 16476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.62:40505 every 8 connection(s)
I20260928 15:07:41.705278 16409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/data/info.pb
I20260928 15:07:41.705574 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 16409
I20260928 15:07:41.705701 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-0/wal/instance
I20260928 15:07:41.705901 26092 external_mini_cluster.cc:1428] Setting key bbdcbef0128de2b1d10645be1528957c
I20260928 15:07:41.706465 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:41.708845 16477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.123.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.61@KRBTEST.COM" created.
Sep 28 15:07:41 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:41.723248 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.709880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:40505 (local address 127.0.0.1:48128)
0928 15:07:41.710072 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.710123 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:41.710636 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:41.710999 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:41.711001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:41.711309 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:41.711477 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.711481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.713138 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.713139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.713888 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.713891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.714604 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.714605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.714666 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.715280 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:41.715289 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:41.718975 (+ 3686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:41.721508 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.721516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.721519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.721746 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.722047 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.722048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.722048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.722148 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.722727 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:41.722734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:41.722801 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 15:07:41.722895 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:07:41.723047 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260928 15:07:41.723359 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.709915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.25.123.62:40505)
0928 15:07:41.710122 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:41.710133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:41.710242 (+ 109us) server_negotiation.cc:407] Connection header received
0928 15:07:41.710807 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:41.710809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:41.710836 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:41.710972 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:41.711587 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.712996 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.713986 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.714480 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.714570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.719122 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:41.719131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:41.719132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:41.719140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:41.721385 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.721842 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.721846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.721846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.721968 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.722241 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.722243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.722243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.722615 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:41.722711 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:41.722961 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 15:07:41.723144 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260928 15:07:41.724335 16477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:41.724987 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.724828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52420)
0928 15:07:41.724903 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.724960 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:07:41.725143 16426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
I20260928 15:07:41.725589 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.725493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52424)
0928 15:07:41.725540 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.725565 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 15:07:41.725696 16477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:41653: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab.
Entry for principal kudu/127.25.123.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab.
I20260928 15:07:41.736838 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:41653
--webserver_interface=127.25.123.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab
--principal=kudu/127.25.123.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:40505,127.25.123.61:41653,127.25.123.60:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:07:41.774984 16477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } attempt: 1
I20260928 15:07:41.775615 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.775441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52432)
0928 15:07:41.775527 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.775583 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:07:41.776070 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.775943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52446)
0928 15:07:41.776012 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.776045 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:07:41.776207 16477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:41653: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
W20260928 15:07:41.823994 16484 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:07:41.824262 16484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:41.824296 16484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:41.824343 16484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:41.825560 16484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:41.825637 16484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:41.825661 16484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:41.825683 16484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:41.825713 16484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:41.827236 16484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:40505,127.25.123.61:41653,127.25.123.60:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.61.keytab
--principal=kudu/127.25.123.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.61:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.123.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:41.827526 16484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:41.827773 16484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:41.832777 16489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:41.832777 16490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:41.832913 16492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:41.833232 16484 server_base.cc:1034] running on GCE node
I20260928 15:07:41.833510 16484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:41.833819 16484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:41.835004 16484 hybrid_clock.cc:648] HybridClock initialized: now 1790608061834984 us; error 39 us; skew 500 ppm
Sep 28 15:07:41 dist-test-slave-qnz0 krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:41.837707 16484 init.cc:377] Logged in from keytab as kudu/127.25.123.61@KRBTEST.COM (short username kudu)
I20260928 15:07:41.838218 16484 webserver.cc:460] Webserver started at http://127.25.123.61:45667/ using document root <none> and password file <none>
I20260928 15:07:41.838474 16484 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:41.838521 16484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:41.838663 16484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:41.839754 16484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data/instance:
uuid: "24a285674f0b42d3b61116cabd623349"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-qnz0"
server_key: "8026f5f1703568eb033396f2dd39b3ff"
server_key_iv: "1c1df9f88c9c3a1f79ba293b4625da79"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.840124 16484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/wal/instance:
uuid: "24a285674f0b42d3b61116cabd623349"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-qnz0"
server_key: "8026f5f1703568eb033396f2dd39b3ff"
server_key_iv: "1c1df9f88c9c3a1f79ba293b4625da79"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.841586 16484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 15:07:41.842439 16499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:41.842682 16484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:41.842758 16484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/wal
uuid: "24a285674f0b42d3b61116cabd623349"
format_stamp: "Formatted at 2026-09-28 15:07:41 on dist-test-slave-qnz0"
server_key: "8026f5f1703568eb033396f2dd39b3ff"
server_key_iv: "1c1df9f88c9c3a1f79ba293b4625da79"
server_key_version: "encryptionkey@0"
I20260928 15:07:41.842868 16484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:41.844370 16477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } attempt: 2
I20260928 15:07:41.844910 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.844745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52452)
0928 15:07:41.844829 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.844864 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 15:07:41.845327 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.845239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52456)
0928 15:07:41.845282 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.845309 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 15:07:41.845439 16477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.61:41653: Network error: Client connection negotiation failed: client connection to 127.25.123.61:41653: connect: Connection refused (error 111)
I20260928 15:07:41.859576 16484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:41.862059 16484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:41.862187 16484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:41.866873 16484 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.61:41653
I20260928 15:07:41.866905 16551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.61:41653 every 8 connection(s)
I20260928 15:07:41.867290 16484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/data/info.pb
I20260928 15:07:41.868443 16552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:41.870033 16552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:41.872494 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 16484
I20260928 15:07:41.872596 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-1/wal/instance
I20260928 15:07:41.872790 26092 external_mini_cluster.cc:1428] Setting key aa0cdfdb5a1f42c12919bcd8f71399d5
Sep 28 15:07:41 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.123.60@KRBTEST.COM; defaulting to no policy
I20260928 15:07:41.880594 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.870809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.25.123.62:40505)
0928 15:07:41.870904 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:41.870906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:41.871087 (+ 181us) server_negotiation.cc:407] Connection header received
0928 15:07:41.871572 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:41.871573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:41.871587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:41.871654 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:41.872307 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.873051 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.874283 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.874541 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.874606 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.877421 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:41.877423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:41.877423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:41.877431 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:41.879035 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.879493 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.879495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.879495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.879564 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.879847 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.879848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.879848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.880010 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:41.880068 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:41.880378 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 15:07:41.880493 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:07:41.880787 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.870772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:40505 (local address 127.0.0.1:48136)
0928 15:07:41.870953 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.870996 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:41.871452 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:41.871730 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:41.871733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:41.871999 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:41.872199 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.872202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.873190 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.873192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.874179 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.874182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.874667 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.874669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.874834 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.875466 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:41.875478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:41.877278 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:41.879149 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.879158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.879161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.879412 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.879644 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.879645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.879645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.879772 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.880072 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:41.880079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:41.880237 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 15:07:41.880346 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:07:41.880534 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260928 15:07:41.881440 16552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.123.60@KRBTEST.COM" created.
Sep 28 15:07:41 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:41.889093 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.881728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52468)
0928 15:07:41.881788 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.881799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:41.881843 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:41.882119 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:41.882120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:41.882276 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:41.882336 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.882338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.883247 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.883248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.883764 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.883766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.884267 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.884268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.884332 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.884843 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:41.884848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:41.886097 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:41.887976 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.887977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.887978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.888142 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.888369 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.888369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.888370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.888447 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.888724 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:41.888725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:41.888773 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 15:07:41.888841 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 15:07:41.888949 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 15:07:41.889158 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.881795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.25.123.61:41653)
0928 15:07:41.881985 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:41.881989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:41.882006 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:07:41.882023 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:41.882024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:41.882035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:41.882121 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:41.882414 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.883159 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.883857 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.884172 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.884271 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.886198 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:41.886206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:41.886206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:41.886213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:41.887869 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.888237 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.888240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.888240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.888299 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.888506 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.888507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.888507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.888653 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:41.888745 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:41.888882 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 15:07:41.889022 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
I20260928 15:07:41.889761 16552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:41.890208 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.890066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55226)
0928 15:07:41.890124 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.890178 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 15:07:41.890367 16502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
I20260928 15:07:41.890851 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.890724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55240)
0928 15:07:41.890797 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.890828 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:07:41.890969 16552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:33221: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.123.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab.
Entry for principal kudu/127.25.123.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab.
I20260928 15:07:41.901101 26092 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGgw5f9/build/release/bin/kudu
/tmp/dist-test-taskGgw5f9/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:33221
--webserver_interface=127.25.123.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab
--principal=kudu/127.25.123.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.123.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.123.62:40505,127.25.123.61:41653,127.25.123.60:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:07:41.936630 16477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } attempt: 3
I20260928 15:07:41.940239 16552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } attempt: 1
I20260928 15:07:41.940719 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.940557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55256)
0928 15:07:41.940636 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.940681 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 15:07:41.941099 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.940984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55270)
0928 15:07:41.941048 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.941077 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 15:07:41.941191 16552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:33221: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Sep 28 15:07:41 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:41.945497 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.937259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52472)
0928 15:07:41.937330 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.937343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:41.937430 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:41.937608 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:41.937609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:41.937794 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:41.937863 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.937865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.938562 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.938563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.939102 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:41.939103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.939531 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.939532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:41.939622 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.940136 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:41.940142 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:41.941736 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:41.943921 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.943923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.943924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.944381 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.944714 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:41.944715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:41.944715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:41.944797 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:41.945137 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:41.945138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:41.945187 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 15:07:41.945248 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 15:07:41.945421 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 15:07:41.945497 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.937350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.25.123.61:41653)
0928 15:07:41.937434 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:41.937436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:41.937449 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:07:41.937509 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:41.937515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:41.937528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:41.937589 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:41.937962 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.938468 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.939208 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:41.939450 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:41.939534 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:41.941832 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:41.941833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:41.941834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:41.941841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:41.943719 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.944539 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.944541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.944542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.944615 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:41.944909 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:41.944910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:41.944910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:41.945061 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:41.945136 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:41.945320 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:07:41.945406 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:07:41.946414 16477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:41.946862 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.946728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55276)
0928 15:07:41.946788 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.946829 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 15:07:41.947373 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.947209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55292)
0928 15:07:41.947241 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.947341 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 15:07:41.947649 16477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:33221: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
I20260928 15:07:41.978884 16477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } attempt: 1
I20260928 15:07:41.979358 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.979225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55300)
0928 15:07:41.979285 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.979319 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:07:41.979759 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:41.979676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55314)
0928 15:07:41.979714 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:41.979736 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 15:07:41.979848 16477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:33221: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
W20260928 15:07:41.992468 16559 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:07:41.992751 16559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:07:41.992770 16559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:07:41.992795 16559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:07:41.994015 16559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:07:41.994060 16559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:07:41.994076 16559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:07:41.994088 16559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:07:41.994100 16559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:07:41.995671 16559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.123.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.123.62:40505,127.25.123.61:41653,127.25.123.60:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/krb5kdc/kudu_127.25.123.60.keytab
--principal=kudu/127.25.123.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.123.60:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.123.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 14:04:12 UTC on ccycloud.quasar-xcuqxv.root.comops.site
build id 4677369
I20260928 15:07:41.995966 16559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:07:41.996351 16559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:07:42.001852 16564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:42.001909 16559 server_base.cc:1034] running on GCE node
W20260928 15:07:42.001876 16565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:07:42.001876 16567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:07:42.002503 16559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:07:42.002872 16559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:07:42.004078 16559 hybrid_clock.cc:648] HybridClock initialized: now 1790608062004049 us; error 42 us; skew 500 ppm
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608062, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:07:42.007268 16559 init.cc:377] Logged in from keytab as kudu/127.25.123.60@KRBTEST.COM (short username kudu)
I20260928 15:07:42.008178 16559 webserver.cc:460] Webserver started at http://127.25.123.60:39183/ using document root <none> and password file <none>
I20260928 15:07:42.008553 16559 fs_manager.cc:359] Metadata directory not provided
I20260928 15:07:42.008631 16559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:07:42.008800 16559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:07:42.009358 16552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } attempt: 2
I20260928 15:07:42.009996 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.009778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55330)
0928 15:07:42.009916 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:42.009958 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 15:07:42.010473 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.010344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55334)
0928 15:07:42.010403 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:42.010437 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 15:07:42.010599 16552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.123.60:33221: Network error: Client connection negotiation failed: client connection to 127.25.123.60:33221: connect: Connection refused (error 111)
I20260928 15:07:42.010512 16559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data/instance:
uuid: "f96779bf7c8c4d309357b11b817fff68"
format_stamp: "Formatted at 2026-09-28 15:07:42 on dist-test-slave-qnz0"
server_key: "50874a9a933e66a85b20750bd707e1ab"
server_key_iv: "730e320f13c94eb06ef5777b9d99edfc"
server_key_version: "encryptionkey@0"
I20260928 15:07:42.010983 16559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/wal/instance:
uuid: "f96779bf7c8c4d309357b11b817fff68"
format_stamp: "Formatted at 2026-09-28 15:07:42 on dist-test-slave-qnz0"
server_key: "50874a9a933e66a85b20750bd707e1ab"
server_key_iv: "730e320f13c94eb06ef5777b9d99edfc"
server_key_version: "encryptionkey@0"
I20260928 15:07:42.012595 16559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 15:07:42.013571 16574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:07:42.013870 16559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 15:07:42.013978 16559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data,/tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/wal
uuid: "f96779bf7c8c4d309357b11b817fff68"
format_stamp: "Formatted at 2026-09-28 15:07:42 on dist-test-slave-qnz0"
server_key: "50874a9a933e66a85b20750bd707e1ab"
server_key_iv: "730e320f13c94eb06ef5777b9d99edfc"
server_key_version: "encryptionkey@0"
I20260928 15:07:42.014043 16559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:07:42.034209 16559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:07:42.036916 16559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:07:42.037050 16559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:07:42.041618 16559 rpc_server.cc:307] RPC server started. Bound to: 127.25.123.60:33221
I20260928 15:07:42.041706 16626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.123.60:33221 every 8 connection(s)
I20260928 15:07:42.042054 16559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/data/info.pb
I20260928 15:07:42.043183 16627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:07:42.044817 16627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:42.046799 26092 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGgw5f9/build/release/bin/kudu as pid 16559
I20260928 15:07:42.046909 26092 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGgw5f9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790607748072058-26092-0/minicluster-data/master-2/wal/instance
I20260928 15:07:42.047063 26092 external_mini_cluster.cc:1428] Setting key 7aad60b0b9144c82710a5f21fd2dcb81
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608062, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:07:42.054996 16477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } attempt: 2
I20260928 15:07:42.057785 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.045685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.25.123.62:40505)
0928 15:07:42.045785 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.045787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.046016 (+ 229us) server_negotiation.cc:407] Connection header received
0928 15:07:42.046573 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.046574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.046586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.046655 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.048209 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.048987 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.050464 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.050718 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.050798 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.053639 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.053641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.053642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.053651 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.056313 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.056751 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.056752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.056753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.056812 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.057059 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.057060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.057060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.057170 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.057225 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.057579 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.057687 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:07:42.057952 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.045599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.62:40505 (local address 127.0.0.1:48142)
0928 15:07:42.045884 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:42.045931 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:42.046437 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:42.047326 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:42.047329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:42.047751 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:42.048083 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.048090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.049104 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.049105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.050350 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.050353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.050832 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.050833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.050934 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.051604 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:42.051615 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:42.053473 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:42.056419 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.056427 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.056429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.056668 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.056879 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.056879 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.056880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.056987 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.057239 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:42.057244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:42.057415 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 15:07:42.057533 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:07:42.057701 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":170,"thread_start_us":133,"threads_started":1}
I20260928 15:07:42.058876 16627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:42.059026 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.047493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.25.123.60:33221)
0928 15:07:42.047637 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.047641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.047679 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:07:42.047746 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.047747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.047771 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.047841 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.048628 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.049617 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.050664 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.051064 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.051120 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.054051 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.054059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.054060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.054067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.056000 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.056497 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.056500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.056501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.056589 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.057701 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.057702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.057702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.058482 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.058633 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.058723 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.058805 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.62@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608062, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:42.064440 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.055306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55344)
0928 15:07:42.055436 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:42.055450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:42.055498 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:42.055700 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:42.055701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:42.055906 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:42.055961 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.055963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.056814 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.056815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.057596 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.057598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.058156 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.058157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.058221 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.058951 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:42.058956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:42.060566 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:42.063122 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.063124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.063124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.063358 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.063627 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.063628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.063628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.063689 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.064051 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:42.064052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:42.064100 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 15:07:42.064154 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 15:07:42.064269 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 15:07:42.065131 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.055360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.25.123.60:33221)
0928 15:07:42.055556 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.055558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.055566 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:07:42.055587 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.055588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.055599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.055673 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.056047 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.056753 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.057693 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.058060 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.058146 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.060661 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.060663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.060663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.060670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.063036 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.063467 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.063468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.063468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.063530 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.063816 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.063817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.063817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.063959 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.064811 (+ 852us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.064880 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.064967 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":118,"threads_started":1}
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.62@KRBTEST.COM
I20260928 15:07:42.066951 16477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73: Bootstrap starting.
I20260928 15:07:42.067528 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.059566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.25.123.61:41653)
0928 15:07:42.059623 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.059626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.059633 (+ 7us) server_negotiation.cc:407] Connection header received
0928 15:07:42.059659 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.059660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.059672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.059731 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.060060 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.060792 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.061639 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.061869 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.061947 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.064037 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.064038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.064039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.064047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.066327 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.066710 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.066712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.066712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.066778 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.066985 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.066986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.066987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.067111 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.067198 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.067353 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.067434 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 15:07:42.067528 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.059372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.61:41653 (local address 127.0.0.1:52486)
0928 15:07:42.059416 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:42.059427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:42.059462 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:42.059738 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:42.059739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:42.059892 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:42.059941 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.059943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.060908 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.060910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.061523 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.061525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.061955 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.061956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.062029 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.062552 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:42.062557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:42.063940 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:42.066429 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.066430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.066430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.066609 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.066840 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.066841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.066841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.066907 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.067189 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:42.067190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:42.067238 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 15:07:42.067326 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 15:07:42.067433 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 15:07:42.067794 16477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:42.068148 16477 log.cc:824] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:42.068843 16627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:07:42.069108 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.060009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.25.123.62:40505)
0928 15:07:42.060151 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.060153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.060159 (+ 6us) server_negotiation.cc:407] Connection header received
0928 15:07:42.060177 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.060177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.060186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.060239 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.060894 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.061584 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.062625 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.062943 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.063009 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.065953 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.065954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.065954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.065962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.067625 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.067929 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.067930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.067931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.067989 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.068166 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.068166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.068167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.068297 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.068494 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.068938 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.069022 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 15:07:42.069500 16477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73: No bootstrap required, opened a new log
I20260928 15:07:42.071336 16477 raft_consensus.cc:348] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.071455 16477 raft_consensus.cc:729] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c480d253c1e445a5919d44aa4e9c6d73, State: Initialized, Role: FOLLOWER
I20260928 15:07:42.071513 16477 consensus_queue.cc:260] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.071791 16634 sys_catalog.cc:455] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.071903 16477 sys_catalog.cc:564] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:42.071928 16634 sys_catalog.cc:458] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608062, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.60@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
W20260928 15:07:42.074074 16645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:42.074677 16645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:42.077162 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.069151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55346)
0928 15:07:42.069197 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:42.069208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:42.069243 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:42.069392 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:42.069393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:42.069546 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:42.069593 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.069595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.070309 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.070310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.071076 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.071079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.071649 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.071650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.071715 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.072205 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:42.072210 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:42.074078 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:42.076063 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.076064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.076065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.076263 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.076542 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.076542 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.076543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.076595 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.076854 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:42.076855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:42.076905 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 15:07:42.076966 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 15:07:42.077064 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 15:07:42.077713 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.069212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.25.123.60:33221)
0928 15:07:42.069251 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.069253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.069286 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:07:42.069305 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.069306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.069316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.069371 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.069735 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.070220 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.071201 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.071554 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.071631 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.074174 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.074176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.074176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.074183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.075975 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.076382 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.076383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.076383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.076475 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.076654 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.076655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.076655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.076773 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.076833 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.077546 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.077625 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.123.61@KRBTEST.COM
I20260928 15:07:42.076148 16409 master_runner.cc:186] Error getting master registration for 127.25.123.61:41653: OK, instance_id { permanent_uuid: "24a285674f0b42d3b61116cabd623349" instance_seqno: 1790608061865623 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:07:42.079569 16409 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:07:42.081471 16627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: Bootstrap starting.
I20260928 15:07:42.082381 16627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:42.082715 16627 log.cc:824] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:42.082836 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.072914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.25.123.61:41653)
0928 15:07:42.072969 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.072982 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.073129 (+ 147us) server_negotiation.cc:407] Connection header received
0928 15:07:42.073158 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.073159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.073171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.073216 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.073893 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.074536 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.075897 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.076215 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.076278 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.079217 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.079219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.079220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.079229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.081253 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.081730 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.081731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.081732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.081804 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.082042 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.082043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.082043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.082189 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.082397 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.082590 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.082688 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:07:42.084110 16627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: No bootstrap required, opened a new log
I20260928 15:07:42.084569 26092 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:07:42.085484 16451 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c480d253c1e445a5919d44aa4e9c6d73"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48156
I20260928 15:07:42.085630 16451 raft_consensus.cc:482] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:07:42.085675 16451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:42.086410 16627 raft_consensus.cc:348] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.086575 16627 raft_consensus.cc:729] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96779bf7c8c4d309357b11b817fff68, State: Initialized, Role: FOLLOWER
I20260928 15:07:42.086659 16451 raft_consensus.cc:504] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.086666 16627 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.086906 16451 leader_election.cc:288] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [CANDIDATE]: Term 1 election: Requested vote from peers 24a285674f0b42d3b61116cabd623349 (127.25.123.61:41653), f96779bf7c8c4d309357b11b817fff68 (127.25.123.60:33221)
I20260928 15:07:42.086983 16648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.087162 16648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:42.087218 16600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c480d253c1e445a5919d44aa4e9c6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f96779bf7c8c4d309357b11b817fff68"
I20260928 15:07:42.087325 16600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:42.088819 16600 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c480d253c1e445a5919d44aa4e9c6d73 in term 1.
I20260928 15:07:42.089120 16627 sys_catalog.cc:564] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:42.089175 16427 leader_election.cc:302] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c480d253c1e445a5919d44aa4e9c6d73, f96779bf7c8c4d309357b11b817fff68; no voters:
I20260928 15:07:42.089321 16634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:07:42.089516 16634 raft_consensus.cc:686] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [term 1 LEADER]: Becoming Leader. State: Replica: c480d253c1e445a5919d44aa4e9c6d73, State: Running, Role: LEADER
I20260928 15:07:42.089623 16634 consensus_queue.cc:237] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.091753 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c480d253c1e445a5919d44aa4e9c6d73. Latest consensus state: current_term: 1 leader_uuid: "c480d253c1e445a5919d44aa4e9c6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.091893 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: This master's current role is: LEADER
W20260928 15:07:42.091946 16668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:07:42.092011 16668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:42.093411 16559 master_runner.cc:186] Error getting master registration for 127.25.123.61:41653: OK, instance_id { permanent_uuid: "24a285674f0b42d3b61116cabd623349" instance_seqno: 1790608061865623 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:07:42.096022 16559 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:07:42.098061 16670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:07:42.099164 16670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:07:42.101734 16552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } attempt: 3
I20260928 15:07:42.103551 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.089003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.25.123.60:33221)
0928 15:07:42.089041 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.089043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.089580 (+ 537us) server_negotiation.cc:407] Connection header received
0928 15:07:42.090077 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.090078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.090092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.090155 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.094116 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.094786 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.097936 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.098212 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.098282 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.099617 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.099618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.099618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.099625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.102168 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.102544 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.102545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.102545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.102612 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.102955 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.102956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.102956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.103081 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.103165 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.103366 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.103455 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 15:07:42.103684 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.088641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.25.123.61:41653)
0928 15:07:42.088686 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.088688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.089770 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 15:07:42.090100 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.090101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.090126 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.090161 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.093198 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.093993 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.097278 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.097617 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.097653 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.099549 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.099551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.099551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.099559 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.101357 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.101764 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.101765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.101766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.101825 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.102751 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.102752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.102753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.102897 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.102988 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.103503 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.103578 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 15:07:42.105163 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.088917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.25.123.62:40505)
0928 15:07:42.089014 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.089016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.092721 (+ 3705us) server_negotiation.cc:407] Connection header received
0928 15:07:42.092840 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.092841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.092854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.092888 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.093705 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.094386 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.095450 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.095747 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.095805 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.097516 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.097517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.097518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.097526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.101066 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.104088 (+ 3022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.104090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.104090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.104158 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.104499 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.104500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.104500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.104636 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.104720 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.104976 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.105067 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 15:07:42 dist-test-slave-qnz0 krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790608061, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.123.61@KRBTEST.COM for kudu/127.25.123.60@KRBTEST.COM
I20260928 15:07:42.109031 16600 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [term 1 FOLLOWER]: Refusing update from remote peer c480d253c1e445a5919d44aa4e9c6d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:42.109529 16634 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:42.111011 16648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c480d253c1e445a5919d44aa4e9c6d73. Latest consensus state: current_term: 1 leader_uuid: "c480d253c1e445a5919d44aa4e9c6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.111114 16648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:42.111980 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.102037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.123.60:33221 (local address 127.0.0.1:55366)
0928 15:07:42.102192 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:07:42.102204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 15:07:42.102239 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:07:42.102449 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:07:42.102451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:07:42.102610 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:07:42.102667 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.102668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.104305 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.104306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.105125 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:07:42.105128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.105743 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.105744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:07:42.105814 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.106268 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:07:42.106273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:07:42.107805 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:07:42.110656 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.110658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.110659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.110985 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.111217 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:07:42.111217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:07:42.111218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:07:42.111281 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:07:42.111600 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:07:42.111601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:07:42.111674 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 15:07:42.111752 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:07:42.111879 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260928 15:07:42.112524 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:42.102204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.25.123.60:33221)
0928 15:07:42.102248 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.102250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.102283 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:07:42.102304 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.102305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.102315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.102363 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.102741 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.103541 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.105195 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:42.105563 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:42.105626 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:42.107920 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:07:42.107921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:07:42.107922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:07:42.107930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:07:42.110519 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.111079 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.111080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.111081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.111144 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:07:42.111340 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:07:42.111342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:07:42.111342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:07:42.111511 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:07:42.111585 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:42.111792 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 15:07:42.112022 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 15:07:42.114233 16552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: Bootstrap starting.
I20260928 15:07:42.115051 16552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: Neither blocks nor log segments found. Creating new log.
I20260928 15:07:42.115389 16552 log.cc:824] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: Log is configured to *not* fsync() on all Append() calls
I20260928 15:07:42.117295 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c480d253c1e445a5919d44aa4e9c6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.117408 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:42.117476 16648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c480d253c1e445a5919d44aa4e9c6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.117575 16648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:42.117712 16672 mvcc.cc:205] Tried to move back new op lower bound from 7334330622393470976 to 7334330622320107520. Current Snapshot: MvccSnapshot[applied={T|T < 7334330622393470976}]
I20260928 15:07:42.118976 16552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: No bootstrap required, opened a new log
I20260928 15:07:42.121579 16552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.121789 16552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a285674f0b42d3b61116cabd623349, State: Initialized, Role: FOLLOWER
I20260928 15:07:42.121810 16670 catalog_manager.cc:1347] Generated new cluster ID: 3eddd88c4e9649a2be6e8bc5fe55224a
I20260928 15:07:42.121863 16670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:07:42.121892 16552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } }
I20260928 15:07:42.122210 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.122344 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:42.122359 16552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:07:42.125169 16525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:07:42.126786 16525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [term 1 FOLLOWER]: Refusing update from remote peer c480d253c1e445a5919d44aa4e9c6d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:07:42.127261 16656 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:07:42.127103 16526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c480d253c1e445a5919d44aa4e9c6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24a285674f0b42d3b61116cabd623349"
I20260928 15:07:42.128902 16526 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c480d253c1e445a5919d44aa4e9c6d73 in term 1.
I20260928 15:07:42.131261 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c480d253c1e445a5919d44aa4e9c6d73. Latest consensus state: current_term: 1 leader_uuid: "c480d253c1e445a5919d44aa4e9c6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.131388 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:42.131435 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c480d253c1e445a5919d44aa4e9c6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.131469 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:07:42.131325 16688 catalog_manager.cc:1259] Loaded cluster ID: 3eddd88c4e9649a2be6e8bc5fe55224a
I20260928 15:07:42.140599 16688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: loading cluster ID for follower catalog manager: success
W20260928 15:07:42.141268 16688 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 15:07:42.141359 16688 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:07:42.153573 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c480d253c1e445a5919d44aa4e9c6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c480d253c1e445a5919d44aa4e9c6d73" member_type: VOTER last_known_addr { host: "127.25.123.62" port: 40505 } } peers { permanent_uuid: "24a285674f0b42d3b61116cabd623349" member_type: VOTER last_known_addr { host: "127.25.123.61" port: 41653 } } peers { permanent_uuid: "f96779bf7c8c4d309357b11b817fff68" member_type: VOTER last_known_addr { host: "127.25.123.60" port: 33221 } } }
I20260928 15:07:42.153721 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73 [sys.catalog]: This master's current role is: LEADER
I20260928 15:07:42.156291 16670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:07:42.157177 16670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:07:42.164834 16670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c480d253c1e445a5919d44aa4e9c6d73: Generated new TSK 0
I20260928 15:07:42.165146 16670 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:07:42.201135 16631 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.199483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.25.123.60:33221)
0928 15:07:42.199545 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.199548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.200326 (+ 778us) server_negotiation.cc:407] Connection header received
0928 15:07:42.200503 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.200504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.200517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.200550 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.201055 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55372: BlockingRecv error: recv got EOF from 127.0.0.1:55372 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 15:07:42.202458 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.199738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.25.123.61:41653)
0928 15:07:42.199795 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.199797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.201843 (+ 2046us) server_negotiation.cc:407] Connection header received
0928 15:07:42.201945 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.201947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.201958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.201988 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.202387 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52498: BlockingRecv error: recv got EOF from 127.0.0.1:52498 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 15:07:42.203397 16481 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.200047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.25.123.62:40505)
0928 15:07:42.200088 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.200089 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.202850 (+ 2761us) server_negotiation.cc:407] Connection header received
0928 15:07:42.202950 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.202952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.202963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.202994 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.203328 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48166: BlockingRecv error: recv got EOF from 127.0.0.1:48166 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:07:42.205665 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.204844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.25.123.61:41653)
0928 15:07:42.204884 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.204886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.205127 (+ 241us) server_negotiation.cc:407] Connection header received
0928 15:07:42.205251 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.205252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.205286 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.205317 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.205615 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52500: BlockingRecv error: recv got EOF from 127.0.0.1:52500 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 15:07:42.208858 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.208007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.25.123.61:41653)
0928 15:07:42.208042 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.208044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.208315 (+ 271us) server_negotiation.cc:407] Connection header received
0928 15:07:42.208451 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.208452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.208464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.208495 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.208805 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52504: BlockingRecv error: recv got EOF from 127.0.0.1:52504 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 15:07:42.213335 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.212369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.25.123.61:41653)
0928 15:07:42.212449 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.212451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.212713 (+ 262us) server_negotiation.cc:407] Connection header received
0928 15:07:42.212833 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.212834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.212847 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.212878 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.213277 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52506: BlockingRecv error: recv got EOF from 127.0.0.1:52506 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 15:07:42.220273 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.219193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.25.123.61:41653)
0928 15:07:42.219243 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.219245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.219561 (+ 316us) server_negotiation.cc:407] Connection header received
0928 15:07:42.219686 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.219687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.219701 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.219736 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.220205 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52510: BlockingRecv error: recv got EOF from 127.0.0.1:52510 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 15:07:42.231361 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.230133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.25.123.61:41653)
0928 15:07:42.230187 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.230190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.230508 (+ 318us) server_negotiation.cc:407] Connection header received
0928 15:07:42.230630 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.230631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.230645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.230679 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.231295 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52518: BlockingRecv error: recv got EOF from 127.0.0.1:52518 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 15:07:42.250264 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.249189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.25.123.61:41653)
0928 15:07:42.249239 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.249241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.249558 (+ 317us) server_negotiation.cc:407] Connection header received
0928 15:07:42.249679 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.249680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.249694 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.249728 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.250194 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52522: BlockingRecv error: recv got EOF from 127.0.0.1:52522 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 15:07:42.284951 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.283932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.25.123.61:41653)
0928 15:07:42.283983 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.283985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.284249 (+ 264us) server_negotiation.cc:407] Connection header received
0928 15:07:42.284356 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.284357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.284370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.284445 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.284887 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52524: BlockingRecv error: recv got EOF from 127.0.0.1:52524 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 15:07:42.352121 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.351155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.25.123.61:41653)
0928 15:07:42.351332 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.351335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.351458 (+ 123us) server_negotiation.cc:407] Connection header received
0928 15:07:42.351566 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.351567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.351579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.351655 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.352057 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52532: BlockingRecv error: recv got EOF from 127.0.0.1:52532 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
W20260928 15:07:42.482471 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.481617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.25.123.61:41653)
0928 15:07:42.481695 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.481697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.481708 (+ 11us) server_negotiation.cc:407] Connection header received
0928 15:07:42.481784 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.481786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.481799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.481858 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.482381 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52546: BlockingRecv error: recv got EOF from 127.0.0.1:52546 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 15:07:42.740444 16556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:07:42.739795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.25.123.61:41653)
0928 15:07:42.739870 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:42.739872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:42.739881 (+ 9us) server_negotiation.cc:407] Connection header received
0928 15:07:42.739907 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:42.739908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:42.739918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:42.739984 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:07:42.740343 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52558: BlockingRecv error: recv got EOF from 127.0.0.1:52558 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:07:43.092519 16668 catalog_manager.cc:1259] Loaded cluster ID: 3eddd88c4e9649a2be6e8bc5fe55224a
I20260928 15:07:43.092602 16668 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: loading cluster ID for follower catalog manager: success
I20260928 15:07:43.093592 16668 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: acquiring CA information for follower catalog manager: success
I20260928 15:07:43.093954 16668 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f96779bf7c8c4d309357b11b817fff68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:07:43.142572 16688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: acquiring CA information for follower catalog manager: success
I20260928 15:07:43.143000 16688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24a285674f0b42d3b61116cabd623349: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:07:43.256434 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:43.253743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.25.123.61:41653)
0928 15:07:43.253911 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:43.253914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:43.253922 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:07:43.253958 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:43.253958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:43.254009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:43.254104 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:43.254253 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.254756 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.255549 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.255754 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.255832 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:43.256017 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:43.256128 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:43.256218 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:43.256259 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 15:07:43.256285 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260928 15:07:43.261842 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:43.258955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.25.123.60:33221)
0928 15:07:43.259073 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:43.259074 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:43.259082 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:07:43.259117 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:43.259118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:43.259158 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:43.259232 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:43.259386 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.259862 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.260696 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.260907 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.260972 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:43.261242 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:43.261398 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:43.261498 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:43.261656 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 15:07:43.261717 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260928 15:07:43.266664 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:43.263897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.25.123.60:33221)
0928 15:07:43.263964 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:43.263966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:43.263974 (+ 8us) server_negotiation.cc:407] Connection header received
0928 15:07:43.264056 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:43.264058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:43.264075 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:43.264135 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:43.264315 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.264858 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.265824 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.266130 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.266226 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:43.266344 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:43.266379 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:43.266439 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:43.266537 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 15:07:43.266570 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:07:43.267184 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:43.263784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.25.123.61:41653)
0928 15:07:43.263838 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:43.263840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:43.264118 (+ 278us) server_negotiation.cc:407] Connection header received
0928 15:07:43.264179 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:43.264180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:43.264191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:43.264238 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:43.264480 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.265237 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.266117 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.266443 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.266509 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:43.266645 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:43.266678 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:43.266732 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:43.267073 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 15:07:43.267101 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 15:07:43.267575 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:07:43.263682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.25.123.62:40505)
0928 15:07:43.263815 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 15:07:43.263817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 15:07:43.264004 (+ 187us) server_negotiation.cc:407] Connection header received
0928 15:07:43.264052 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:07:43.264054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:07:43.264107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:07:43.264224 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:07:43.264503 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.265270 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.266248 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:07:43.266566 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:07:43.266666 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:07:43.266955 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:07:43.267262 (+ 307us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:07:43.267360 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:07:43.267410 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 15:07:43.267434 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260928 15:07:43.276968 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 16409
I20260928 15:07:43.283020 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 16484
I20260928 15:07:43.288594 26092 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGgw5f9/build/release/bin/kudu with pid 16559
2026-09-28T15:07:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2922 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5857 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (315215 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST